Writing fields on multiple IDSs might not be thread safe

[Nush395]

Trying to do a write operation to multiple IDS in multiple threads e.g

setattr(core_profiles_ids.profiles_1d.ion[0], 'name', 'D')

results in

AttributeError: type object 'IDSMetadata' has no attribute '__setattr__'. Did you mean: '__delattr__'?

Using imas-core. I believe this is because get_toplevel_metadata() temporarily deletes
IDSMetadata.__setattr__

I was able to avoid the error by adding a lock to get_toplevel_metadata

import threading
from imas import ids_metadata as _ids_metadata
from imas import ids_toplevel as _ids_toplevel

def _patch_imas_metadata_lock() -> None:
  """Patches IMAS metadata construction to be thread-safe.

  The upstream ``get_toplevel_metadata()`` temporarily *deletes*
  ``IDSMetadata.__setattr__`` from the class to allow construction,
  then restores it.  This is not thread-safe.
  """
  _imas_metadata_lock = threading.Lock()
  _orig_get_toplevel_metadata = _ids_metadata.get_toplevel_metadata

  def _thread_safe_get_toplevel_metadata(structure_xml):
    with _imas_metadata_lock:
      return _orig_get_toplevel_metadata(structure_xml)

  _ids_metadata.get_toplevel_metadata = _thread_safe_get_toplevel_metadata
  # Also patch the reference that ids_toplevel already imported.
  _ids_toplevel.get_toplevel_metadata = _thread_safe_get_toplevel_metadata

[maarten-ic]

Nice catch @Nush395! Indeed imas-python wasn't developed with multi-threading in mind, so you might find more issues (perhaps with not-so-clear error messages) when using multiple threads.

Note that IMAS-Core is definitely not thread-safe either, I'd refrain from doing put/get/serialize/deserialize calls from multiple threads to avoid data corruption.

[Nush395]

Hi @maarten-ic, thanks for confirming! Ah ok, so in general is it best to avoid all multi-threaded uses of an IDS or are there some usages that would be ok?

[Nush395]

Also, is enabling multi-thread support something that's on the roadmap at all?

[maarten-ic]

Hi @maarten-ic, thanks for confirming! Ah ok, so in general is it best to avoid all multi-threaded uses of an IDS or are there some usages that would be ok?

I would avoid accessing a single data object (IDS and substructures) simultaneously from multiple threads, and then it might work. This is good practice either way to avoid data races.
However, I cannot guarantee anything, so the safest is indeed to do all IDS handling in a single thread. Since it's all in Python you'll be limited by the GIL (assuming you don't use a free-threaded Python build) and multi-threaded access to IDSs is not going to be faster than single-threaded access.

Also, is enabling multi-thread support something that's on the roadmap at all?

That's a good question. I'm not aware of any initiatives to make IMAS threadsafe, but @olivhoenen may know more.

[olivhoenen]

This topic (among others) is in the back of our mind but not yet in any roadmap at that stage. This year we want first investigate the possibility to use the SWMR model in IMAS-Core, so focusing first on multiple reader processes for IDS data.