Feature/fabrics/caps by jonasohland · Pull Request #33 · jonasohland/mxl

jonasohland · 2026-06-12T16:09:46Z

No description provided.

github-actions · 2026-06-12T16:10:14Z

Test Results

0 tests 0 ✅ 0s ⏱️
0 suites 0 💤
0 files 0 ❌

Results for commit 6763fa6.

github-actions · 2026-06-12T16:10:14Z

Test Results

94 tests +20 94 ✅ +20 12s ⏱️ -1s
1 suites ± 0 0 💤 ± 0
1 files ± 0 0 ❌ ± 0

Results for commit a72e371. ± Comparison against base commit 0ae1dc5.

♻️ This comment has been updated with latest results.

mlefebvre1

I'm not done reviewing yet, but here's what I have so far..

mlefebvre1 · 2026-06-15T01:19:24Z

+            .memoryRegistrationModes = FI_MR_VIRT_ADDR | FI_MR_LOCAL | FI_MR_ALLOCATED | FI_MR_PROV_KEY,
+            .endpointType = FI_EP_MSG,
+            .caps = libfabricCaps(capabilities, isTarget),
+            .supportedAddressFormats = {FI_SOCKADDR_IN, FI_SOCKADDR_IN6},


FI_SOCKADDR_IB ?

I need to test if that actually works and how devices with multiple ports present through libfabric. Right now all providers let you choose the port by specifying the node address. I suspect with raw infiniband addresses, the node address is shared between ports.
I would have to find out how that maps to the node/service scheme.

Yeah, the problem is that the host (or node) portion of FI_SOCKADDR_IB and FI_SOCKADDR_IN6 is the same for interfaces with and auto-assigned ipv6 address.

mlefebvre1 · 2026-06-15T14:51:31Z

+            .supportedAddressFormats = {FI_SOCKADDR_IN, FI_SOCKADDR_IN6},
+            .supportedProtocols = {FI_PROTO_SOCK_TCP},
+            .requiredCaps = libfabricRequiredCaps(capabilities) | FI_MSG,
+            .filteredCaps = FI_TAGGED,


Nitpick, but I don't think it's possible to have FI_TAGGED support for FI_EP_MSG`

When pull() returned no sample and the next batch time had already passed, mxlGetNsUntilIndex() returned 0, making the next pull() non-blocking. With no sleep on the empty-pull path this caused the loop to spin. Clamp the pull timeout to at least one batch period so the thread always blocks between retries. Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

…info Endpoint::Id is a process-local dispatch key -- an initiator routes connection and completion events to the correct target by this id (RCInitiator's id-keyed _targets map). randomId() reseeded a fresh std::mt19937 from a single 32-bit, time-correlated std::random_device draw on EVERY call, so targets set up in the same tight window (a gateway bringing up many mirrors at once) could draw the same seed -> the same id; the initiator's map then collapsed two endpoints and one mirror never connected. Intermittent 'stuck forever' wedge that scaled with concurrent setups (never with 1, ~1/3 with 9). - Endpoint::randomId(): seed one 64-bit engine ONCE from full-width entropy and serialize draws. Collision-free within and across processes; setup/reconnect only, never on the hot path. - FabricInfoList::get(): serialize fi_getinfo -- libfabric does lazy global provider init on first call; concurrent first calls can race it. Signed-off-by: Dominik Heinrich <d.heinrich@qvest-digital.com>

- condense the randomId/fi_getinfo comments to the essentials - adopt the auto-left declaration style (mu, lock, getInfoMutex, getInfoLock) - correct the fi_getinfo comment: fi_ini is mutex-protected upstream, so serialize the unlocked per-provider getinfo loop (defensive, setup-only) Signed-off-by: Dominik Heinrich <d.heinrich@qvest-digital.com>

Address review: fi_ini() is already guarded by libfabric's common_locks.ini_lock and fi_getinfo is documented as thread-safe, so the extra serialization in FabricInfoList::get was guarding a non-bug. Remove the mutex and revert FabricInfo.cpp to baseline. Also simplify the endpoint random engine to a single mt19937_64 seeded from std::random_device. The static mutex still guards concurrent generation; only the redundant seed_seq scaffolding is dropped. Signed-off-by: Dominik Heinrich <d.heinrich@qvest-digital.com>

Signed-off-by: Vincent Trussart <vincent.trussart@grassvalley.com>

Co-authored-by: Michael Lefebvre <michael.lefebvre.31@gmai..com> Signed-off-by: Michael Lefebvre <michael.lefebvre@riedel.net>

A target receives grains by repeatedly calling mxlFabricsTargetReadGrainNonBlocking(), which reads one completion per call. The completion queue has a bounded depth, so on a high-frame-rate or many-stream receiver whose per-grain work is slower than the grain inter-arrival time, completions can accumulate and overflow the queue. On connection-oriented providers (verbs) a CQ overflow surfaces as a fatal queue pair error that tears the connection down. Two changes: - Document the two-phase drain pattern in docs/Fabrics.md section 11: a hot phase that drains all currently-available completions cheaply into an application queue, and a warm phase that performs the bounded heavy per-grain work. Includes the overflow failure mode and completion-queue sizing guidance. - Make the target completion queue depth configurable. The depth was previously hardcoded to 8 (with a 'this should be parameterized' note). It can now be set per target via the existing, previously-unused 'options' argument of mxlFabricsTargetSetup, e.g. {"cqDepth": 256}. This mirrors the JSON options mechanism already used by the Flow API. The value is threaded through TargetWrapper::setup into RCTarget/RDMTarget; 0 or an absent option selects the implementation default, so existing callers are unaffected. cqDepth applies to targets and is ignored for initiators. - Add tests covering a valid cqDepth option, the default (no option), and rejection of invalid options (zero, wrong type, malformed JSON) with MXL_ERR_INVALID_ARG. Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Introduce mxlFabricsTargetReadGrainsNonBlocking() which drains up to a caller-specified number of available grain completions in a single call, returning the drained grain indices and count. This lets receivers empty the completion queue promptly (phase 1) before doing the heavier per-grain commit work (phase 2), instead of looping the single-grain API by hand. - fabrics.h: declare the new batch drain entry point and document its contract (MXL_STATUS_OK if >=1 drained, MXL_ERR_NOT_READY if none, MXL_ERR_INVALID_ARG on bad args). - fabrics.cpp: implement it over the OFI TargetWrapper by looping readGrain(), which also drives connection/EQ progress. - demo.cpp: convert the target runDiscrete() loop to the two-phase drain. - Fabrics.md: update section 11.2 to show the batch call. - test_basics.cpp: add an end-to-end batch drain test plus invalid-arg coverage. Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Fix clang-format violations flagged by CI in the files touched by this branch (include ordering in fabrics.cpp, member/keyword and comment alignment in the target headers). Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

- Drop mxlFabricsTargetReadGrainsNonBlocking: it duplicated the single-grain accessor without real batching (fi_cq_read count stays 1), so the two-phase drain is now documented against the existing API instead. - Introduce an internal TargetSetupOptions struct carrying an std::optional<size_t> cqDepth, replacing the size_t=0 sentinel across the Target/RCTarget/RDMTarget setup() calls. - Revert the demo to the simple single-grain proxy loop and add a comment pointing to the drain-pattern docs. Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

- Replace bare throw with Exception::invalidState in Target/Initiator - Restore commented-out error status check in try_run - Remove duplicate copyStringProperty call for link_type - Remove spurious blank line in Address::fromBase64 - Remove unused FabricSourceInterfaces files and getSources declaration - Remove unused ProviderConfig.hpp includes, add missing include in FabricInterfaceProbe.hpp - Fill in empty doxygen stubs, add missing [[nodiscard]] attributes - Rename capabilitiesDummyCheck to validateCapabilities - Report hostname instead of opaque fi_shm address for SHM interfaces - Propagate service field in SHM interface descriptions - Handle unparseable query addresses gracefully in probeInterfaces Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Separate how capability flags behave on the two code paths: - Query (mxlFabricsGetInterfaces): caps are output-only, not used for filtering. Each returned interface reports its detected capabilities. - Setup (target/initiator): caps express requirements. At least one transfer capability must be set; defaults to REMOTE_WRITE with a warning if missing. Move capability normalization from ProviderConfig into selectSourceInterface so it only applies on the setup path. Derive libfabric caps and requiredCaps from MXL capability flags via helpers, supporting both REMOTE_WRITE and SEND_RECEIVE. Store the resolved ProviderCapabilities in ProviderConfig for later use. Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

…selection Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

…InfoList Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Co-Authored-By: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Co-authored-by: Jonas Ohland <jonas.ohland@gmai.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

jonasohland assigned mlefebvre1 Jun 12, 2026

mlefebvre1 reviewed Jun 12, 2026

View reviewed changes

jonasohland force-pushed the feature/fabrics/caps branch from be8e7a1 to 0bb0699 Compare June 14, 2026 14:36

mlefebvre1 reviewed Jun 15, 2026

View reviewed changes

jonasohland force-pushed the feature/fabrics/caps branch 2 times, most recently from f5f57fd to 2e2ad13 Compare June 17, 2026 11:11

jonasohland and others added 21 commits June 23, 2026 09:36

clang-format all the things

d3771a4

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Add some timings clarifications based on NAB learnings.

d02ad6b

Signed-off-by: Vincent Trussart <vincent.trussart@grassvalley.com>

Reword according to review

35013b9

Signed-off-by: Vincent Trussart <vincent.trussart@grassvalley.com>

PR Review comments

4746c58

Signed-off-by: Vincent Trussart <vincent.trussart@grassvalley.com>

Bump crossbeam-epoch to 0.9.20 to fix RUSTSEC-2026-0204.

7367c74

Co-authored-by: Michael Lefebvre <michael.lefebvre.31@gmai..com> Signed-off-by: Michael Lefebvre <michael.lefebvre@riedel.net>

Apply clang-format

b7ba6d5

Fix clang-format violations flagged by CI in the files touched by this branch (include ordering in fabrics.cpp, member/keyword and comment alignment in the target headers). Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Move parseCqDepthOption into ofi anonymous namespace

d35380b

Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Thread TargetSetupOptions through RCTarget; fix in_options doc

5f30608

Signed-off-by: Maciej Oleksy <maciej.oleksy@intel.com>

Refactor interface fabric info handling, add interface query api

b8b9fc2

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Refactor demo to use mxlFabricsGetInterfaces for automatic interface …

3ece1c7

…selection Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Add documentation to ProviderConfig, FabricInterfaceProbe, and Fabric…

d84ccdb

…InfoList Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Add documentation and maxMessageSize warning

89a6835

Co-Authored-By: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

jonasohland and others added 23 commits July 10, 2026 14:20

Add some test to cover the parts that can be tested without hardware

223770f

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix EFA provider config and use capability flags for blocking decisions

71193a7

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Add missing license headers

cc76a1c

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Correctly clang-format all testing files

a375f44

Co-authored-by: Jonas Ohland <jonas.ohland@gmai.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Upgrade libfabric to the most recent version

cf25bc5

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Add FI_RMA cap unconditionally

0c73c42

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Update mxlFabricsGetInterfaces to use mxlFabricsInstance

6f41418

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Address review comments

e8e9fae

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Use auto with right-side type initialization consistently

acb6cf1

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Add missing brackets around boolean sub-expressions

241075a

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

mxl-fabrics-info: add non-verbose output variant and make it default

09b449d

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Use correct verbs protocol

5704917

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix ProviderConfig filtering by filteredCaps

6159810

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix doc comment for MXL_FABRICS_PROVIDER_ANY.

fc08047

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix checking for missing required caps in ProviderConfig

20d6119

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix segfault in mxl-fabrics-demo on exit due to uninitialized mem read

16c858b

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

mxl-fabrics-info: fix incorrect picojson include

2e188eb

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix service string extraction in FabricAddress

7815fdf

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Introduce strongly typed FabricAddressType to replace ofis uint32_t

8014709

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

address related classes: use nodiscard also for static functions

925e9c1

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Address.{ch}pp : Address style comments

c318681

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Optimize caps string building

4202d4c

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Address mostly style related comments

6f0bf17

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

jonasohland force-pushed the feature/fabrics/caps branch from 6aa56dc to 6f0bf17 Compare July 10, 2026 14:27

jonasohland and others added 5 commits July 10, 2026 17:18

Fix interface enumeration failing for unsupported info objects

aacb0ab

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Enforce required transfer mode caps

b9f5cd0

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Fix tests

777acbb

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Update comment about required transfer caps

6c7fda6

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Address remaining review comments in dmf-mxl#564

a72e371

Co-authored-by: Jonas Ohland <jonas.ohland@gmail.com> Signed-off-by: Jonas Ohland (Riedel) <jonas.ohland@riedel.net>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/fabrics/caps#33

Feature/fabrics/caps#33
jonasohland wants to merge 49 commits into
mainfrom
feature/fabrics/caps

jonasohland commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

mlefebvre1 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlefebvre1 Jun 15, 2026

Uh oh!

jonasohland Jun 15, 2026

Uh oh!

jonasohland Jun 15, 2026

Uh oh!

Uh oh!

mlefebvre1 Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jonasohland commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Test Results

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

mlefebvre1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlefebvre1 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

jonasohland Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

jonasohland Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlefebvre1 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions Bot commented Jun 12, 2026 •

edited

Loading