[Question]: Is efa-dp-direct integration planned as a GPU-initiated transport for libfabric/EFA?

[quanta42]

Question

In issue #4, @a-szegel pointed to amzn/efa-dp-direct as the EFA equivalent of IBGDA, a CUDA library that lets GPU kernels post work requests and poll completions directly on EFA queue pairs without CPU proxy involvement.

Currently, NVSHMEM's libfabric transport on EFA routes all device-initiated operations through a CPU proxy thread. The performance docs note that IBGDA "differs from other transports that rely on passing messages to a CPU proxy thread to initiate the transfer." For fine-grained P2P workloads this proxy path is a severe bottleneck.

Our use case: Ring attention for distributed video diffusion training on AWS P5en (H200, 4× EFA NICs/node). The workload performs nvshmemx_putmem_nbi_on_stream + nvshmemx_signal_op across nodes (32 layers × 7 ring hops × forward+backward).

Observed:
NVSHMEM cross-node ring: ~248 s/step, 1% GPU utilization
Same topology with NCCL P2P: ~65 s/step — 3.8× faster
GPU shows burst-then-idle pattern: 100% during compute, then 0% while proxy drains transfers
This matches the Perseus paper (arXiv:2605.00686) analysis of proxy fence serialization reducing signaled transfer throughput to ~2% of unsignaled at high concurrency on non-IBGDA fabrics.

Questions:
Is there a plan to integrate efa-dp-direct as a GPU-initiated transport in NVSHMEM (similar to how IBGDA provides GPU-direct NIC submission for Mellanox)?

If not planned, are there alternative API patterns or tuning parameters that can reduce the proxy serialization overhead for fine-grained signaled P2P on EFA?

Environment:
NVSHMEM v3.6.5-0 (with patches #35, #76)
CUDA 12.8.1, NCCL v2.29.7, aws-ofi-nccl v1.19.0, GDRCopy v2.5.1
4× H200 nodes (P5en), 8 GPUs/node, 4× EFA NICs/node

[amitrad-aws]

Thanks for the detailed report and the reference to the Perseus paper.

To answer your questions,
There is no current plan to integrate efa-dp-direct to the libfabric transport.
The libfabric transport is designed to be fabric-agnostic through the OFI abstraction. Integrating efa-dp-direct would mean bypassing libfabric entirely and adding a hardware-specific GPU-initiated path for a single provider, which defeats the purpose of the transport's portability.

As for the alternative, the proxy isn't actually the problem here.
The Perseus paper you reference actually arrives at this conclusion — the performance collapse comes from fence-induced serialization in the ordering mechanism between put and signal, not from the proxy submission model itself.
This is addressed in the upcoming NVSHMEM 3.7 release.
We've made significant optimizations to both the libfabric transport and the proxy in general, including removing the need for a fence between put and signal.

I'd recommend re-benchmarking your ring attention workload on 3.7 when it's available.

[quanta42]

Thanks for the detailed report and the reference to the Perseus paper.

To answer your questions, There is no current plan to integrate efa-dp-direct to the libfabric transport. The libfabric transport is designed to be fabric-agnostic through the OFI abstraction. Integrating efa-dp-direct would mean bypassing libfabric entirely and adding a hardware-specific GPU-initiated path for a single provider, which defeats the purpose of the transport's portability.

As for the alternative, the proxy isn't actually the problem here. The Perseus paper you reference actually arrives at this conclusion — the performance collapse comes from fence-induced serialization in the ordering mechanism between put and signal, not from the proxy submission model itself. This is addressed in the upcoming NVSHMEM 3.7 release. We've made significant optimizations to both the libfabric transport and the proxy in general, including removing the need for a fence between put and signal.

I'd recommend re-benchmarking your ring attention workload on 3.7 when it's available.

Thank you, the fence removal in 3.7 is great news and should address the dominant bottleneck we observed. We'll definitely re-benchmark once it's available.

One clarification on "the proxy isn't actually the problem here." We agree the fence serialization is the primary issue for our current workload. But architecturally, the proxy path is still fundamentally different from IBGDA: a single CPU thread submitting to the NIC vs hundreds of GPU threads posting work queue entries in parallel. IBGDA has no proxy at all. Even with the fence removed, the proxy's single-threaded submission rate will be a ceiling that doesn't exist on IBGDA — whether that ceiling matters in practice depends on message rates and transfer sizes, which is why benchmarking on 3.7 is the right next step.

Two follow-ups:
Is there a timeline or early access path for 3.7? We'd like to test ring attention and fused FSDP (AllGather+GEMM overlap with per-tile signaling) as soon as possible.

Does the optimization apply to putmem_signal_nbi specifically, or to the broader put + signal pattern across all libfabric providers?

Best,

[amitrad-aws]

I understand your concern about the message rate with the proxy, there are limitations with creating a GDA path for libfabric that we need to see the real advantage to see if we should tackle it.
The optimization is broader and applies for put + signal/amo across all libfabric providers.

[benhg]

Hi @quanta42, NVSHMEM 3.7 has been released.

[quanta42]

Hi @quanta42, NVSHMEM 3.7 has been released.

@benhg @amitrad-aws, thanks for the heads up, we re-benchmarked.

Setup: Same topology as the original report: 8 × H200 (P5en, 16× EFAv3 NICs/node), 64 GPUs, ring=16 cross-node EFA, 1080p 10s V2V training, SAC-ON.

Environment: CUDA 13.2.1, NCCL v2.30.4, NVSHMEM v3.7.0-0, FI_EFA_ENABLE_SHM_TRANSFER=0 (per 3.7 release notes).

Transport	NVSHMEM ver.	Step time	vs NCCL P2P
NVSHMEM symm_mem=true	3.6.5	~248 s/step	3.8×
NVSHMEM symm_mem=true	3.7.0	~175 s/step	2.7×
NCCL P2P symm_mem=false	—	~65 s/step	1.0×

The fence removal and AWS-contributed libfabric optimizations (RMA batching, signaling, ack aggregation) delivered a ~30% improvement, significant and appreciated.

However, a 2.7× gap to NCCL P2P remains on the same hardware and topology.

We also tested the new 3.7 NIC/affinity knobs:

• NVSHMEM_CPU_AFFINITY: Defaults to AUTO (NUMA-local pinning already active), confirmed no change.

• NVSHMEM_NETDEVS_POLICY=EXTERNAL_SHARING_PCIE_SWITCH_NIC_EXCLUSIVE with NVSHMEM_ENABLE_NIC_PE_MAPPING=1: Actually regressed to ~260 s/step (worse than 3.6.5). The default AUTO policy, which balances across local PEs, is clearly better for our 16-NIC P5en topology.

• NVSHMEM_LIBFABRIC_PROXY_REQUEST_BATCH_MAX=256 (8× default of 32): No change (~175 s/step). The proxy drains requests fast enough; queue depth isn't the limiter.

• RMA batching: Already on by default (NVSHMEM_LIBFABRIC_DISABLE_BATCH_RMA=false). Confirmed no extra tuning needed.

Are there other env vars or tuning recommendations we should try to close the remaining gap?

[erezzarum]

Hi @quanta42, NVSHMEM 3.7 has been released.

@benhg @amitrad-aws, thanks for the heads up, we re-benchmarked.

Setup: Same topology as the original report: 8 × H200 (P5en, 16× EFAv3 NICs/node), 64 GPUs, ring=16 cross-node EFA, 1080p 10s V2V training, SAC-ON.

Environment: CUDA 13.2.1, NCCL v2.30.4, NVSHMEM v3.7.0-0, FI_EFA_ENABLE_SHM_TRANSFER=0 (per 3.7 release notes).

Transport NVSHMEM ver. Step time vs NCCL P2P
NVSHMEM symm_mem=true 3.6.5 ~248 s/step 3.8×
NVSHMEM symm_mem=true 3.7.0 ~175 s/step 2.7×
NCCL P2P symm_mem=false — ~65 s/step 1.0×
The fence removal and AWS-contributed libfabric optimizations (RMA batching, signaling, ack aggregation) delivered a ~30% improvement, significant and appreciated.

However, a 2.7× gap to NCCL P2P remains on the same hardware and topology.

We also tested the new 3.7 NIC/affinity knobs:

• NVSHMEM_CPU_AFFINITY: Defaults to AUTO (NUMA-local pinning already active), confirmed no change.
• NVSHMEM_NETDEVS_POLICY=EXTERNAL_SHARING_PCIE_SWITCH_NIC_EXCLUSIVE with NVSHMEM_ENABLE_NIC_PE_MAPPING=1: Actually regressed to ~260 s/step (worse than 3.6.5). The default AUTO policy, which balances across local PEs, is clearly better for our 16-NIC P5en topology.
• NVSHMEM_LIBFABRIC_PROXY_REQUEST_BATCH_MAX=256 (8× default of 32): No change (~175 s/step). The proxy drains requests fast enough; queue depth isn't the limiter.
• RMA batching: Already on by default (NVSHMEM_LIBFABRIC_DISABLE_BATCH_RMA=false). Confirmed no extra tuning needed.

Are there other env vars or tuning recommendations we should try to close the remaining gap?

Hey, can you please share the benchmark you are running to reproduce it? Thanks!

[quanta42]

Hi @quanta42, NVSHMEM 3.7 has been released.

@benhg @amitrad-aws, thanks for the heads up, we re-benchmarked.
Setup: Same topology as the original report: 8 × H200 (P5en, 16× EFAv3 NICs/node), 64 GPUs, ring=16 cross-node EFA, 1080p 10s V2V training, SAC-ON.
Environment: CUDA 13.2.1, NCCL v2.30.4, NVSHMEM v3.7.0-0, FI_EFA_ENABLE_SHM_TRANSFER=0 (per 3.7 release notes).
Transport NVSHMEM ver. Step time vs NCCL P2P
NVSHMEM symm_mem=true 3.6.5 ~248 s/step 3.8×
NVSHMEM symm_mem=true 3.7.0 ~175 s/step 2.7×
NCCL P2P symm_mem=false — ~65 s/step 1.0×
The fence removal and AWS-contributed libfabric optimizations (RMA batching, signaling, ack aggregation) delivered a ~30% improvement, significant and appreciated.
However, a 2.7× gap to NCCL P2P remains on the same hardware and topology.
We also tested the new 3.7 NIC/affinity knobs:

• NVSHMEM_CPU_AFFINITY: Defaults to AUTO (NUMA-local pinning already active), confirmed no change.
• NVSHMEM_NETDEVS_POLICY=EXTERNAL_SHARING_PCIE_SWITCH_NIC_EXCLUSIVE with NVSHMEM_ENABLE_NIC_PE_MAPPING=1: Actually regressed to ~260 s/step (worse than 3.6.5). The default AUTO policy, which balances across local PEs, is clearly better for our 16-NIC P5en topology.
• NVSHMEM_LIBFABRIC_PROXY_REQUEST_BATCH_MAX=256 (8× default of 32): No change (~175 s/step). The proxy drains requests fast enough; queue depth isn't the limiter.
• RMA batching: Already on by default (NVSHMEM_LIBFABRIC_DISABLE_BATCH_RMA=false). Confirmed no extra tuning needed.

Are there other env vars or tuning recommendations we should try to close the remaining gap?

Hey, can you please share the benchmark you are running to reproduce it? Thanks!

Yes. This is our production ring-attention training benchmark, not a standalone NVSHMEM perftest binary.

The workload is our 1080p / 10-second video-to-video training benchmark using ring attention:

• 8 × P5en/H200 nodes, 64 GPUs total
• 8 GPUs per node, 16× EFAv3 NICs per node
• ring attention group size is 16 GPUs, so each ring spans 2 nodes and crosses EFA
• SAC-ON means selective activation checkpointing is enabled
• symm_mem=true selects the NVSHMEM/libfabric/EFA implementation for ring-attention P2P
• symm_mem=false keeps the same ring-attention topology but uses NCCL P2P instead

The relevant communication pattern is ring-style multi-PE put -> quiet/completion -> signal -> poll, repeated across layers and forward/backward. So raw shmem_put_bw is not representative, and even 2-PE shmem_put_signal_ping_pong_latency may miss the proxy/quiet pressure we see at 64 GPUs.

The closest public benchmark I found is the AWS Labs NVSHMEM perftest harness:
https://github.com/awslabs/awsome-distributed-ai/tree/main/micro-benchmarks/nvshmem

That looks useful as a baseline, but the README still appears to reference NVSHMEM 3.3.9 and standard perftests such as shmem_put_bw, shmem_put_latency, shmem_put_signal_ping_pong_latency, and collectives. I do not see a 3.7.0-specific ring-style put/quiet/signal/poll benchmark there.

From our steady-state nsys trace, the remaining gap looks dominated by completion/proxy behavior rather than raw RMA bandwidth: nvshmemi_proxy_quiet_entrypoint() is the largest NVSHMEM-specific GPU cost, while the RMA/signal kernels themselves are comparatively small. We can share the trace details if useful.

So the closest minimal repro would be a ring-style benchmark where each PE repeatedly:

1. issues NVSHMEM PUTs to the next peer / ring neighbor,
2. performs stream-ordered quiet/completion before publishing readiness,
3. signals the remote peer,
4. polls for incoming readiness from the previous peer,
5. repeats this across many layers/iterations and many PEs.

Could you share what benchmark NVIDIA/AWS used to validate the NVSHMEM 3.7 EFA improvements? In particular, was it run with NVSHMEM 3.7.x on P5en/H200, and did it include put/quiet/signal/poll behavior under multi-PE load?