Add mlx5 DC (Dynamically Connected) transport support

[harshavardhana]

Summary

Add support for mlx5 Dynamically Connected (DC) transport, which enables scalable RDMA without per-peer QP setup. DC is critical for large clusters where RC (Reliable Connection) creates N² QP state.

Motivation

With RC, each node needs a dedicated QP per remote connection. At 100+ nodes with 64 connections each, this becomes 6400+ QPs per node. DC reduces this to 1 DCT (target) + W DCI (initiator) QPs per node, regardless of cluster size.

What DC provides

• DCI (DC Initiator): Client-side QP that can talk to any DCT on any remote node
• DCT (DC Target): Server-side QP that accepts from any DCI
• Per-WR addressing: wr_set_dc_addr(ah, remote_dctn, dc_key) sets the target per work request, not per QP
• dc_key: 64-bit shared secret for authentication

Required API additions

rdma-mummy-sys (C bindings)

• Bind mlx5dv.h — specifically:
  • mlx5dv_create_qp() for DCI/DCT creation
  • mlx5dv_context and mlx5dv_query_device() for capability detection
  • MLX5DV_QP_INIT_ATTR_MASK_DC, MLX5DV_DCTYPE_DCT, MLX5DV_DCTYPE_DCI
  • DC-related fields in mlx5dv_qp_init_attr
• Link against libmlx5 provider library

sideway (Rust API)

• Mlx5DeviceContext — query DC capabilities
• DcInitiatorQueuePair / DcTargetQueuePair types via mlx5dv_create_qp()
• wr_set_dc_addr(ah, dctn, dc_key) on the extended send path
• QueuePairType::DcInitiator / QueuePairType::DcTarget enum variants
• Feature-gated: #[cfg(feature = "mlx5")] since DC is Mellanox/NVIDIA-specific

Hardware tested on

• ConnectX-6 (vendor_part_id 4129), firmware 28.44.1036
• MLNX_OFED 25.01, rdma-core 2501mlnx56
• mlx5dv.h present with MLX5DV_QP_INIT_ATTR_MASK_DC confirmed

References

• RDMA DC Transport (Mellanox docs)
• mlx5dv.h API

Happy to contribute a PR for this. Can test on real ConnectX-6 hardware.

[dragonJACson]

We’ve also wanted to add wrappers for direct verbs for some time, but since there hasn’t been strong demand, we’ve kept it at a relatively low priority.

That said, thank you very much for raising this use case and for working on a PR. We really appreciate it. Before moving forward, though, we think some discussion is needed to make the implementation more elegant and maintainable, especially to reduce code duplication.

For mlx5dv, we’d like to get the design right from the beginning. If we establish a solid foundation now, it should make it much easier to support other direct verbs interfaces in the future, such as efadv and hnsdv.

Our preferred approach is to first provide a dlopen-based static library for mlx5 in rdma-core-mummy, then add the bindings in rdma-mummy-sys, and finally implement the higher-level support in sideway.

I’ll leave additional comments in the corresponding PRs with more detailed feedback.