./nvbandwidth -t device_to_device_memcpy_read_ce failed due to unknown issue

[plorrrrrrr]

I execute ./nvbandwidth -t device_to_device_memcpy_read_ce

However, it output as follows:

$ ./nvbandwidth -t device_to_device_memcpy_read_ce  
nvbandwidth Version: v0.7
Built from Git version: v0.7

CUDA Runtime Version: 12040
CUDA Driver Version: 12040
Driver Version: 550.127.08

Device 0: NVIDIA A100-SXM4-80GB (00000000:34:00)
Device 1: NVIDIA A100-SXM4-80GB (00000000:35:00)
Device 2: NVIDIA A100-SXM4-80GB (00000000:36:00)
Device 3: NVIDIA A100-SXM4-80GB (00000000:37:00)
Device 4: NVIDIA A100-SXM4-80GB (00000000:9b:00)
Device 5: NVIDIA A100-SXM4-80GB (00000000:9c:00)
Device 6: NVIDIA A100-SXM4-80GB (00000000:9d:00)
Device 7: NVIDIA A100-SXM4-80GB (00000000:9e:00)

Running device_to_device_memcpy_read_ce.
 Invalid value when checking the pattern at 0x70a640000000
 Current offset : 11136 
ASSERT in expression h_errorFlag == 0 in void memcmpPatternHelper(CUstream, CUdeviceptr, long long unsigned int, unsigned int, CompareKernelFunc, std::shared_ptr<NodeHelper>)() : /mnt/project/cuda_related/nvbandwidth/memcpy.cpp:165

[deepakcu]

Is the issue consistently reproducible? Can you try a lower buffer size? with --bufferSize=64? Do you see the issue with any other test (e.g. ./nvbandwidth -t device_to_device_memcpy_write_ce/read_sm/write_sm) ?
Was nvbandwidth binary compiled on the same system where the test is run?

[gc435]

I don't see this error the first time I run the test, but I regularly see it after that. I have to reboot in order to run it again. Ubuntu 24.04.2, driver 550, CUDA 12.8. nvbandwidthtest was compiled on the same computer.

[jinpuwang]

I hit the same issue with h200:

[root@supportlive nvbandwidth]# ./nvbandwidth 
nvbandwidth Version: v0.8
Built from Git version: v0.8

CUDA Runtime Version: 13000
CUDA Driver Version: 13000
Driver Version: 580.95.05

supportlive
Device 0: NVIDIA H200 NVL (00000000:03:00)
Device 1: NVIDIA H200 NVL (00000000:42:00)
Device 2: NVIDIA H200 NVL (00000000:8c:00)
Device 3: NVIDIA H200 NVL (00000000:c7:00)
Device 4: NVIDIA H200 NVL (00000001:04:00)
Device 5: NVIDIA H200 NVL (00000001:44:00)
Device 6: NVIDIA H200 NVL (00000001:8e:00)
Device 7: NVIDIA H200 NVL (00000001:c8:00)

Running host_to_device_memcpy_ce.
memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     57.63     57.63     57.63     57.63     57.63     57.63     57.63     57.63

SUM host_to_device_memcpy_ce 461.05

Running device_to_host_memcpy_ce.
memcpy CE CPU(row) <- GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     57.31     57.31     57.31     57.31     57.31     57.31     57.31     57.31

SUM device_to_host_memcpy_ce 458.48

Running host_to_device_bidirectional_memcpy_ce.
memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     57.23     57.25     57.33     57.24     57.27     57.37     57.36     57.29

SUM host_to_device_bidirectional_memcpy_ce 458.33

Running device_to_host_bidirectional_memcpy_ce.
memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     36.49     36.88     36.85     36.63     36.71     36.90     36.87     36.51

SUM device_to_host_bidirectional_memcpy_ce 293.83

Running device_to_device_memcpy_read_ce.
 Invalid value when checking the pattern at 0x7f1e20000000
 Current offset : 267040 
ASSERT in expression h_errorFlag == 0 in void memcmpPatternHelper(CUstream, CUdeviceptr, long long unsigned int, unsigned int, CompareKernelFunc, std::shared_ptr<NodeHelper>)() : /root/nvbandwidth/memcpy.cpp:165

[harrym91]

Also hitting this on H200 NVL, rocky9, 590.48.01 drivers. using dgcmi binary download,not compiled from source.