Bootstrap: fixed the issue where modify_qp failed or ah is Null due to a Bootstrap TCP ordering error

[gzshan]

Summary

This PR fixes an intermittent boot-stage failure in the UID boot plugin that causes downstream RDMA setup to fail, accompanied by one of the following errors:

• ibv_modify_qp failure (invalid remote QP attributes), or
• NULL Address Handle (ah == NULL) when creating QP/AH from bootstrap-exchanged data.

Background & Root Cause

Step 1 — Localizing the failure to bootstrap.

In large-scale runs we frequently hit ibv_modify_qp failures (errno 22 / EINVAL). Log analysis showed the QP attributes fed into modify_qp were invalid — specifically, the remote QPN and GID exchanged via bootstrap were zero. By dumping the payload before and after the bootstrap exchange, we confirmed the data was correct before the bootstrap alltoall, but became zero after it. This localized the bug to the bootstrap layer itself, not the RDMA transport.

Step 2 — Confirming cross-step message mis-matching.

The UID bootstrap mechanism establishes a short-lived TCP connection for each message and performs multiplexing and demultiplexing at the receiver based on the (peer, tag) combination. Since successive all-to-all and barrier steps legitimately reuse the same (peer, tag) identifiers, we suspected potential conflicts between these steps. To verify this, we modified the data associated with the barrier operation; in the traces where failures occurred, the bootstrap_recv call within the all-to-all step retrieved a connection carrying the barrier payload. This confirmed that the subsequently initiated barrier connection was received before the in-transit all-to-all connection and was erroneously processed as an all-to-all connection.

Step 3 — Root cause: kernel accept order diverging from wire order.

Tcpdump at the moment of failure showed the SYNs arriving in the expected wire order, but the receiver kernel handed them to accept() in a different order. Under high connection churn, the NIC's RSS hash distributes incoming TCP connections across multiple RX queues, each drained by a different CPU via softirq. When some of those CPUs are momentarily busy, their queues drain more slowly, and a connection that arrived later on the wire can complete the handshake and enter the accept queue earlier than one that arrived before it. Combined with the (peer, tag)-only matching in bootstrap_recv, this reordering silently swaps message payloads between steps — surfacing downstream as zeroed QPN/GID and ibv_modify_qp EINVAL.

Reproduction

We built a minimal standalone reproducer that issues many short-lived TCP connections in a tight burst, mimicking the bootstrap send/recv pattern. Under this workload the receiver's accept() order is easily observed to diverge from the sender's connect() order within seconds — reproducing the exact symptom described above and confirming that the reordering originates in the kernel accept path rather than in NVSHMEM.

Fix

Fix 1 — TX-drain barrier on the sender

Add a TX-drain barrier on the sender, right before close(), so that connection N is guaranteed to be enqueued on the receiver before the next connect()/SYN is issued for connection N+1.

New helper: bootstrap_send_drain_txq()
Reads the socket fd via nccl_fn_table(get_fd, ...).
Polls ioctl(fd, SIOCOUTQ, &unacked) — SIOCOUTQ reports unsent + unacknowledged bytes; it reaches 0 only after the peer has ACKed everything, which is only possible after the peer socket has reached ESTABLISHED, i.e. after this connection is already in the receiver's accept queue.

Sleeps 200µs between polls; bounded by a configurable timeout and by abort_flag.

Fix 2 — Disjoint tag spaces for alltoall and barrier

The reordering above is only observable because bootstrap_recv de-multiplexes purely by (peer, tag). In the original code, bootstrap_uid_alltoall and bootstrap_uid_barrier both started their tag counter from a small value and incremented it per step, so their tag ranges overlapped. If a barrier connection was accepted ahead of an in-flight alltoall connection, the (peer, tag) of the barrier message could collide with what the alltoall recv was waiting for, and the barrier payload would be consumed in place of the alltoall payload (which is exactly the "zeroed QPN/GID" symptom).

New environment variable

Added to env_defs.h:

NVSHMEM_BOOTSTRAP_UID_SEND_DRAIN_MS (int, default: 1000)
> 0 : enable barrier with the given timeout (ms).
<= 0 : disable the barrier entirely (restores the previous behavior).

Thanks

[rkowalewski]

Hi @gzshan, Thanks for the detailed investigation and proposed fix. To help us reproduce and validate the issue, could you please share:

• The standalone reproducer code and instructions for running it
• The number of nodes, PEs, and GPUs used
• The CUDA and NVSHMEM version/commit
• NIC and GPU model
• The OS distribution and Linux kernel version

[gzshan]

@rkowalewski Hi, Thank you for your reply. Regarding this issue, I would like to add the following:

Software versions

• CUDA：12.2 or 13.0
• Driver version: 535.247.01
• Gcc：13.3.0
• nvshmem: We initially discovered this issue in v3.4.5, but it still exists in the latest v3.7.1 and the devel branch. The difference is that in the new version, in bootstrap_net_recv, if recv_size is not equal to size, it will directly throw an error.

OS/kernel

• OS-Distribution：Ubuntu 24.04.3 LTS
• kernel: 5.4.241-1-tlinux4-0017.7

Cluster / run configuration

• Nodes: 2
• PEs (ranks): 16 (8 PEs per node)
• GPUs per node: NVIDIA H20、H800
• Interconnect: NVIDIA ConnectX-7, RoCE v2

It is worth noting that we reproduced this issue while running the DeepEP inter-node tests; therefore, it can be reproduced even when there are only two PEs within the same track.

Standalone reproducer

We reproduce the bug by first driving the receiver into the same "RSS + softirq saturated" state that our production cluster hits, and then running a bootstrap-heavy NVSHMEM workload on top.

The environment is built with two small shell scripts (attached: server_side.sh / client_side.sh):

• server_side.sh runs on the DUT and starts many iperf3 UDP servers on consecutive ports, pinned to the NUMA node that owns the NIC IRQs.

• client_side.sh runs on any peer host and launches the same number of iperf3 -u -l 64 -b 0 clients — a large number of UDP small packets across many 5-tuples, so RSS spreads them over many RX queues and softirq gets saturated on the DUT.

# on the DUT
./server_side.sh 5201 128
# on the peer
./client_side.sh <DUT_IP> 5201 128 600 64 

client_side.sh
server_side.sh
With this background load running, launching a bootstrap-heavy NVSHMEM workload on the DUT reliably reproduces the zeroed remote QPN/GID and ibv_modify_qp EINVAL within seconds; without the load the same workload can run for a long time before hitting it. Applying the fix in this PR eliminates the failure under the same load.

[rkowalewski]

Hi @gzshan, thanks for the detailed investigation. We’ve confirmed the issue and will review the proposed changes internally as we work toward an appropriate solution.