Summary
GPU-Direct RDMA aggregate throughput decreases as more processes on the same host use it. Eight processes together move less than one process does. This is the main thing stopping S3-over-RDMA from being usable for multi-GPU DDP training, where one process per GPU is the norm.
I don't think this is a minio-cpp bug — it looks like it sits in libcuobjclient/libcufile or in the server's per-client RDMA resource handling — but filing here since this is where the client stack lives and you'll know who to route it to.
Measurement
Client: 8× H200, 256 cores, single 400 GbE RoCE rail (iperf3 ceiling 41.5 GB/s).
Server: 2-node AIStor RELEASE.2026-08-07T18-34-35Z, 48 NVMe, EC:4.
minio-cpp: main @ e764b82, built -DMINIO_CPP_ENABLE_RDMA=ON.
Each process runs 8 threads, each thread with its own 768 MiB CUDA destination buffer, reading 768 MiB objects.
| Processes × 8 streams |
Aggregate |
RDMA share of bytes |
| 1 |
27.84 GB/s |
100% |
| 2 |
5.58 GB/s |
– |
| 4 |
3.66 GB/s |
– |
| 8 |
1.91 GB/s |
33% |
At 256 MiB buffers the RDMA share stays 100% at 8 processes and it is still slow, so the fallback and the slowdown are two separate effects:
| Config |
Aggregate |
RDMA share |
| 1 process × 8 streams × 256 MiB |
8.30 GB/s |
100% |
| 8 processes × 8 streams × 256 MiB |
3.94 GB/s |
100% |
Aggregate operation rate collapses too, roughly independently of size: 139 ops/s at 1 process vs 17.5 ops/s at 8 (64 MiB objects).
What I ruled out
- ibverbs resource limits.
max_mr = 16,777,216, max_pd = 8,388,608, max_mr_size unlimited; only ~374 MRs in use.
- cuFile pinned-memory cap. Setting
max_device_pinned_mem_size_kb to 64 GiB explicitly changed nothing.
- Per-transfer registration churn. I prototyped a caller-pinned buffer registry so
cuMemObjGetDescriptor/cuMemObjPutDescriptor are skipped for pre-pinned buffers. It made no reliable difference (median 9.77 vs 10.02 GB/s at 256 MiB — if anything slightly worse), so I dropped it rather than propose it. Registration churn is not the mechanism.
- A crash. Earlier builds segfaulted in this shape;
main does not. This is now purely a throughput/fallback problem.
Note on the silent fallback
At 768 MiB × 8 processes only a third of the bytes were carried by RDMA; the rest completed over HTTP with no error surfaced. Since use_rdma is just cuMemObjGetDescriptor(...) == 0, a registration that fails under pressure is indistinguishable from "RDMA not wanted", and the caller sees a successful transfer that was three times slower than it should have been.
It would help a lot if a declined-because-registration-failed transfer were distinguishable from a normal HTTP fallback — a counter, or a log at debug level. Right now the only way to tell is to diff minio_api_rdma_read_bytes_total on the server across the operation, which is what I ended up doing.
Reproduction
Harness and full raw results (JSON per run) are in a benchmark repo I can share if useful. The shape is: N processes, each with 8 threads, each thread issuing miniocpp_get_object into its own registered CUDA buffer, with the server's minio_api_rdma_read_bytes_total sampled around the run to compute the RDMA share.
Summary
GPU-Direct RDMA aggregate throughput decreases as more processes on the same host use it. Eight processes together move less than one process does. This is the main thing stopping S3-over-RDMA from being usable for multi-GPU DDP training, where one process per GPU is the norm.
I don't think this is a minio-cpp bug — it looks like it sits in
libcuobjclient/libcufileor in the server's per-client RDMA resource handling — but filing here since this is where the client stack lives and you'll know who to route it to.Measurement
Client: 8× H200, 256 cores, single 400 GbE RoCE rail (
iperf3ceiling 41.5 GB/s).Server: 2-node AIStor
RELEASE.2026-08-07T18-34-35Z, 48 NVMe, EC:4.minio-cpp:
main@e764b82, built-DMINIO_CPP_ENABLE_RDMA=ON.Each process runs 8 threads, each thread with its own 768 MiB CUDA destination buffer, reading 768 MiB objects.
At 256 MiB buffers the RDMA share stays 100% at 8 processes and it is still slow, so the fallback and the slowdown are two separate effects:
Aggregate operation rate collapses too, roughly independently of size: 139 ops/s at 1 process vs 17.5 ops/s at 8 (64 MiB objects).
What I ruled out
max_mr= 16,777,216,max_pd= 8,388,608,max_mr_sizeunlimited; only ~374 MRs in use.max_device_pinned_mem_size_kbto 64 GiB explicitly changed nothing.cuMemObjGetDescriptor/cuMemObjPutDescriptorare skipped for pre-pinned buffers. It made no reliable difference (median 9.77 vs 10.02 GB/s at 256 MiB — if anything slightly worse), so I dropped it rather than propose it. Registration churn is not the mechanism.maindoes not. This is now purely a throughput/fallback problem.Note on the silent fallback
At 768 MiB × 8 processes only a third of the bytes were carried by RDMA; the rest completed over HTTP with no error surfaced. Since
use_rdmais justcuMemObjGetDescriptor(...) == 0, a registration that fails under pressure is indistinguishable from "RDMA not wanted", and the caller sees a successful transfer that was three times slower than it should have been.It would help a lot if a declined-because-registration-failed transfer were distinguishable from a normal HTTP fallback — a counter, or a log at debug level. Right now the only way to tell is to diff
minio_api_rdma_read_bytes_totalon the server across the operation, which is what I ended up doing.Reproduction
Harness and full raw results (JSON per run) are in a benchmark repo I can share if useful. The shape is: N processes, each with 8 threads, each thread issuing
miniocpp_get_objectinto its own registered CUDA buffer, with the server'sminio_api_rdma_read_bytes_totalsampled around the run to compute the RDMA share.