transparent hugepages

[pordyna]

Is there any reason why scipp requires setting transparent_hugepage to enabled while other libraries like numpy use the hugepages already at madvise? I observed some very significant performance differences on systems with the madvise setting. Now I usually use scipp on HPC systems where I do not have control over these settings. On some clusters transparent hugepages are disabled, but modules are provided for usage of explicit hugepages. This is fine, but other systems do not provide a solution for using explicit hugepages and keep the default madvise.

[SimonHeybrock]

Good question. I guess you also refer to our docs page: https://scipp.github.io/user-guide/tips-tricks-and-anti-patterns.html#Improving-performance:-Allocators-and-HugePages.

To be honest, I do not remember why madvise was not sufficient (as the note in https://scipp.github.io/user-guide/tips-tricks-and-anti-patterns.html#Using-HugePages-transparently implies). This is worth finding out.

On HPC, check if the system has enabled explicit hugepages, then configure as described on the page linked on top.

[SimonHeybrock]

The answer turned out to be squarely on our side.

Scipp allocates its buffers with a plain new[], i.e., via malloc, and never called madvise(MADV_HUGEPAGE). NumPy does, for every array of 4 MByte and more, which is exactly why its arrays use HugePages with the madvise setting while ours did not. It is not a TBB issue either, though libtbbmalloc_proxy.so does not help: tbbmalloc contains no madvise call at all and treats HugePages as unavailable unless they are reserved explicitly or the setting is always. So the note in our docs was accurate, but for a reason we can fix.

If #3937 pans out and gets merged, Scipp would then by default request HugePages for buffers of 4 MByte and more, matching NumPy's threshold. On the system I measured on (madvise, defrag=madvise, kernel 6.8) that is worth 1.47x for an elementwise operation and 1.66x for hist, on 1 GByte buffers. Thanks for pushing on this, the effect is larger than we assumed.

Until this is released you can get the same effect with any current version, without root, provided the cluster has glibc 2.35 or newer:

$ GLIBC_TUNABLES=glibc.malloc.hugetlb=1 python ...

This makes glibc itself advise the memory it hands out, which covers all of Scipp's buffers, not just the large ones. On older glibc the tunable does not exist, so there the fix above is the only option — if that is the situation on your clusters, let us know, since it would be an argument for a patch release.

In case you want to check your own systems, the script and the raw measurements are here: https://gist.github.com/SimonHeybrock/132292e373059e68f6ac970524caeb23

[pordyna]

Thanks for looking into it and providing a workaround. It looks like my python interpreter is linked against gilibc 2.35 so the workaround should be fine. I will have a look. I'm sometimes also using python/scipp inside an apptainer container I guess the GLIBC_TUNABLES would have to be applied inside the container?

So yes it can be quite slow. So as an example I was loading data from disk, doing some light processing and saving to disk, (ca. 500GiB in 64 chunks). When allocating a buffer with sc.empty for each chunk and directly loading the data into it the whole process took over 40 min. When letting the IO library do the initial allocation and copying the loaded buffer into scipp with sc.array(values=buffer, ..) it was only 15min even with the extra copy.

[SimonHeybrock]

I'm sometimes also using python/scipp inside an apptainer container I guess the GLIBC_TUNABLES would have to be applied inside the container?

Never used apptainer. I use VScode devcontainers (which user docker), and there hugepages were simply not available (I don't know if it could be configured somehow).

So yes it can be quite slow. So as an example I was loading data from disk, doing some light processing and saving to disk, (ca. 500GiB in 64 chunks). When allocating a buffer with sc.empty for each chunk and directly loading the data into it the whole process took over 40 min. When letting the IO library do the initial allocation and copying the loaded buffer into scipp with sc.array(values=buffer, ..) it was only 15min even with the extra copy.

Some things to note on:

• sc.empty should be fast -- it does not initialize memory.
• How are you copying into sc.empty?
• The copy in sc.array(values=buffer, ..) should be multi-threaded (including the part that is claiming pages), so that might be the reason why it is fast.
• Remember that you will typically see big differences between the first time your process claims memory (and releases it again) and subsequent times (because free() keeps around the pages, only marks them as available).

[MridulS]

Also fwiw scipp 26.7.0 is out which includes #3937, maybe you could test it out?

[pordyna]

Thanks for bringing the patch out so fast! I will test it in and out of a container.

Some things to note on:

* `sc.empty` should be fast -- it does not initialize memory.

* How are you copying into `sc.empty`?

So the copy into sc.empty is done by our I/O library and in this particular configuration should have be done with an ADIOS2 backend which is by default multithreaded, but I am quite sure we are bound by disk/network speed here.

When I use sc.empty I pass it to https://github.com/openPMD/openPMD-api/blob/043030b6cf34add375aa27076e032c944ae8ca05/src/binding/python/RecordComponent.cpp#L687 where it expects a pybind py::buffer. While when I let the openpmd-api allocate first it is calling the py::array constructor ( and I think this is just calling numpy's array constructor) in https://github.com/openPMD/openPMD-api/blob/043030b6cf34add375aa27076e032c944ae8ca05/src/binding/python/RecordComponent.cpp#L806 and passing this as the buffer to the previous specialisation.