Add opt-in contiguous grain pool with single memory registration

[moleksy]

Discrete-flow grains are stored as one shared-memory mapping per grain, so mapping every grain individually for device DMA (e.g. DPDK/VFIO) consumes one memseg entry per grain and can exhaust the bounded memseg list under multi-stream load (flows x grains). The same one-per-grain cost applies to the libfabric memory registrations the Fabrics layer performs for RDMA.

Add an opt-in contiguous grain pool and have the Fabrics layer take advantage of it:

Grain pool (mxl-internal):

• When a discrete flow is created with the "grainPool": true option, all of its grains are stored back-to-back at a fixed stride (sizeof(Grain) + grainPayloadSize) in a single backing file (grains/pool.data) instead of one file per grain (grains/data.{i}). The grains are then contiguous in virtual memory.
• DiscreteFlowData gains a pool storage mode (openGrainPool/isPoolMode/ poolBase/poolMappedSize/grainStride); grainAt()/grainCount() are pool-aware. The per-grain file path is unchanged and remains the default.
• FlowOptionsParser parses the boolean "grainPool" option; Instance forwards it to FlowManager::createOrOpenDiscreteFlow (new defaulted parameter, so existing callers are unaffected).
• The writer creates the pool and initialises each grain header; readers auto-detect the pool layout by the presence of grains/pool.data.

Single memory registration (mxl-fabrics):

• Domain::registerRegions now detects when the regions are contiguous in host memory and registers them as a single fi_mr_regattr instead of one registration per region. RegisteredRegion holds a shared MemoryRegion plus its offset within that registration.
• Each grain's LocalRegion/RemoteRegion is unchanged in addr/len and the RMA ring-slot indexing (localRegions[grainIndex % grainCount]) still holds; all grains share one desc/rkey. For non-contiguous regions (per-grain files) the behaviour is unchanged (one registration each).
• This drops the number of NIC memory registrations per flow from grainCount to one when the pool is used.

[jonasohland]

Hello! Thanks for your PR. First of all, I would like to properly understand the limitations of the current implementation. In most of my testing, for reasonable history sizes, I ran out of bandwidth before running out of memory registration resources. So what numbers are we looking at in your use case?

Also:
For proper offloaded RMA, we have to register the memory on both sides. But the initiator (which is a logical MXL flow reader) has no control over how the grain storage is created. If we decide to accept this PR, my suggestion would be to instead move this option to the domain level (where the history size already lives) so the creator of the domain can influence this directly, or maybe even replace the way grains are created entirely with the approach from this PR.

[vt-tv]

I can see the benefits of this appoach. one of the reasons why we implemented individual grains as separate files is to help the memory management system allocate smaller pages. Don't we risk hitting situations where the memory subsystem is unable to allocate such large contiguous regions of memory when operating in UHD 59p on an OS that has been running for a very long uptime? (in other words, memory is fragmented). Or is this a non issue?

[moleksy]

Hi Jonas - thank you for your review. I'd say that both points are fair.

On the limitations - resource we actually do exhaust is on the other side of the same grain memory: the DPDK/VFIO DMA mapping. In our demo that we showed at NAB the NIC DMAs ST2110 packets directly into MXL grain payloads (zero-copy), so every grain has to be mtl_dma_map'd into the IOMMU / DPDK external-memseg list. With per-grain files each grain is an independent shm mmap, hence one external-memory registration per grain, and DPDK caps those at RTE_MAX_MEMSEG_LISTS = 128. So 16 streams × 8 grains = 128 registrations, which hits the ceiling immediately, regardless of resolution.

Coalescing collapses each flow's grains into a single registration (in this case 8 to 1), so the same fan-out drops from 128 lists to 16. So the win we care about here is "let's not blow the whole DPDK memseg / VFIO mapping budget when fanning out many low-history flows," and the single-MR effect on the fabrics side just falls out of the same contiguous layout for free.

As for moving this option to domain level, I think this would be a good approach.

On "replace grain creation entirely with this approach" - sure, I'm open to that. It would mean one layout, no auto-detection branch, no per-grain-file path. I basically wanted to make it opt-in to ensure backward compatibility with the existing design.

Hello! Thanks for your PR. First of all, I would like to properly understand the limitations of the current implementation. In most of my testing, for reasonable history sizes, I ran out of bandwidth before running out of memory registration resources. So what numbers are we looking at in your use case?

Also: For proper offloaded RMA, we have to register the memory on both sides. But the initiator (which is a logical MXL flow reader) has no control over how the grain storage is created. If we decide to accept this PR, my suggestion would be to instead move this option to the domain level (where the history size already lives) so the creator of the domain can influence this directly, or maybe even replace the way grains are created entirely with the approach from this PR.

[moleksy]

Hello Vincent,

In my opinion no physical continuity is required here, only virtual. The pool is a single mmap of one tmpfs file with no MAP_POPULATE/MAP_HUGETLB, so pages still fault in lazily as ordinary 4 KiB pages, just as they do today with per-grain files. The kernel never has to find one large contiguous physical block, the pool can be (and after long uptime surely will be just as you mentioned) scattered across physical frames. So physical fragmentation is irrelevant to whether the mapping succeeds.

All that changes here is virtual layout: N VMAs to 1 VMA. Finding a contiguous virtual hole is pretty easy considering we have like 128 TiB of user VA, and the per-grain split never actually changed the page size the MM hands out - both layouts are 4 KiB tmpfs pages. So I'd say that this is a non-issue.

I can see the benefits of this appoach. one of the reasons why we implemented individual grains as separate files is to help the memory management system allocate smaller pages. Don't we risk hitting situations where the memory subsystem is unable to allocate such large contiguous regions of memory when operating in UHD 59p on an OS that has been running for a very long uptime? (in other words, memory is fragmented). Or is this a non issue?

[garethsb]

Large physical pages and/or physical contiguity are beneficial, right?

[moleksy]

Yes, overall contiguity/huge pages are beneficial (for TLB reach and especially for DMA), and the pool layout doesn't sacrifice that.

This proposal doesn't exactly make sure that we have physical contiguity - my point in reply to Vincent was that fragmentation can't cause the mapping to fail, because nothing here forces huge pages or contiguity.

On the other hand, the pool actually helps rather than hurts, because a single contiguous virtual mapping over one tmpfs file is the layout that Transparent Huge Pages (and, if we eventually want it, an explicit hugetlbfs backing) can collapse into 2 MiB pages, on the other side - N separate per-grain mappings are far less likely to be promoted.

So if anything, coalescing makes the huge-page/contiguity win more attainable, not less. The per-grain split is making the large physical pages and/or physical contiguity less attainable.

Large physical pages and/or physical contiguity are beneficial, right?

[garethsb]

I agree with both things, beneficial now, but that perhaps we ought to be exposing the ability to require 2MiB hugepages.

[jonasohland]

Hi Jonas - thank you for your review. I'd say that both points are fair.

On the limitations - resource we actually do exhaust is on the other side of the same grain memory: the DPDK/VFIO DMA mapping. In our demo that we showed at NAB the NIC DMAs ST2110 packets directly into MXL grain payloads (zero-copy), so every grain has to be mtl_dma_map'd into the IOMMU / DPDK external-memseg list. With per-grain files each grain is an independent shm mmap, hence one external-memory registration per grain, and DPDK caps those at RTE_MAX_MEMSEG_LISTS = 128. So 16 streams × 8 grains = 128 registrations, which hits the ceiling immediately, regardless of resolution.

Coalescing collapses each flow's grains into a single registration (in this case 8 to 1), so the same fan-out drops from 128 lists to 16. So the win we care about here is "let's not blow the whole DPDK memseg / VFIO mapping budget when fanning out many low-history flows," and the single-MR effect on the fabrics side just falls out of the same contiguous layout for free.

As for moving this option to domain level, I think this would be a good approach.

On "replace grain creation entirely with this approach" - sure, I'm open to that. It would mean one layout, no auto-detection branch, no per-grain-file path. I basically wanted to make it opt-in to ensure backward compatibility with the existing design.

Hello! Thanks for your PR. First of all, I would like to properly understand the limitations of the current implementation. In most of my testing, for reasonable history sizes, I ran out of bandwidth before running out of memory registration resources. So what numbers are we looking at in your use case?
Also: For proper offloaded RMA, we have to register the memory on both sides. But the initiator (which is a logical MXL flow reader) has no control over how the grain storage is created. If we decide to accept this PR, my suggestion would be to instead move this option to the domain level (where the history size already lives) so the creator of the domain can influence this directly, or maybe even replace the way grains are created entirely with the approach from this PR.

Thanks! I was not at NAB and did not know about your demo. But that sounds like real limitation and it would be great if MXL could support your usecase.

[jonasohland]

I am going to open an issue for this so we can put it on the TSC meeting agenda.

[KimonHoffmann]

Only touching on a peripheral point of the discussion here:

As for moving this option to domain level, I think this would be a good approach.

I think "domain level options" in general are a flawed concept (just like a the already existing history duration domain-level option), because the domain of a media function may be a cherry picked collection of bind-mounted flows that are gathered from other physical locations, each of which may have its own customized "domain level" options, effectively resulting in all options that are supposed to be equal across the entire domain being different for each of the flows.

[jonasohland]

Only touching on a peripheral point of the discussion here:

As for moving this option to domain level, I think this would be a good approach.

I think "domain level options" in general are a flawed concept (just like a the already existing history duration domain-level option), because the domain of a media function may be a cherry picked collection of bind-mounted flows that are gathered from other physical locations, each of which may have its own customized "domain level" options, effectively resulting in all options that are supposed to be equal across the entire domain being different for each of the flows.

I don't think that this is at all relevant here. "Domain level options" are just a way for the orchestration layer to inform a media function that wants to publish something to a directory to use a certain storage layout or history size. That does not really imply any deeper meaning of the domain concept.

[KimonHoffmann]

What I mean is that they're suggestive and may lead consumers to assume things that aren't guaranteed or enforceable.

I agree though that they're an appropriate means for an orchestration layer to provide defaults to producers.