[Question]: send in reduce-scatter VS directsend in all-reduce

[GG-yuki]

Question

I find a interesting problem that, I thought all-reduce and reduce-scatter has the same (k - 1) steps before.
However, reduce-scatter just use send and recv, with allreduce use directsend and directrecv.
https://github.com/NVIDIA/nccl/blob/master/src/device/all_reduce.h#L48
https://github.com/NVIDIA/nccl/blob/master/src/device/reduce_scatter.h#L42

I wonder what's the difference? Or, why cause the difference? Is it related to performance?
PS: gpt-5 told me that reduce-scatter has less steps, so it can save the time of pointer conversion. Is it right?

[sjeaugey]

Reductions do not really benefit from having access to the remote buffer, as we need to receive in the intermediate buffer before we perform the reduction. So, there is no point in using directSend/directRecv for reduction operations.

Allreduce fuses reducescatter and allgather operations, so it makes sense for allreduce to leverage the direct access in the allgather phase.

[GG-yuki]

@sjeaugey Kind of you for reply. I totally agree that in allgather need direct.
I got that reduction don't need save the intermediate result, and that's what reduce-scatter did.
So, the point is, why all-reduce need direct buffer before step k - 1, and that is reduce step, different from reduce-scatter. I found that in nccl 2.19, it is just like reduce-scatter.
https://github.com/NVIDIA/nccl/blob/v2.19/src/device/all_reduce.h#L64
So why in 2.29 version it changes to direct in reduce step? Does it improve the performance or some other advantages?

[sjeaugey]

Oh I see.

It was just part of a larger change to allow for direct pointer access when registering buffers, changing a lot of send/recv to directSend/directRecv. But indeed, in this precise case, I don't think it was useful to change it. But maybe it didn't have any effect.

@KaimingOuyang please keep me honest in case I'm missing the point.