Harmonize AD paths with data management

[canismarko]

Writing area detector files to DM storage in ophyd-async

Problem description

How do we save area detector files in a structure that is compatible
with data management permissions on Sojournor?

Data management's access model expects a folder to correspond to a
data management experiment, then regular POSIX permissions are used to
restrict access.

How do we structure area detector file storage to match this model?

Right now, for ophyd async area detectors we're using the path provider protocol
to automatically set the path. We need a path provider to run an
ophyd-async area detector, so I think the task is: write a path
provider that includes data management support.

Option 1: Each day is a DM experiment

We could leave the structure as is and consider each day to be a DM
experiment.

Pros:

• Easy

Cons:

• Access control must be done exclusively by Tiled
• Doesn't really solve the problem, just ignores it

Here is how we are currently using the path provider:

• define a function default_path_provider() that:
  • loads the global iconfig.toml file
  • looks for a key "area_detector_root_path"
  • Creates a YMDPathProvider with the UUIDFilenameProvider using
    the root path
• Each area detector includes the argument path_provider to __init__()
  • If this argument is omitted, default_path_provider() is called

# File: path_provider.py
from collections.abc import Mapping
from pathlib import Path

from ophyd_async.core import UUIDFilenameProvider, YMDPathProvider

from ..._iconfig import load_config

def default_path_provider(path: Path | None = None, config: Mapping | None = None):
    if config is None:
        config = load_config()
    if path is None:
        path = Path(config.get("area_detector_root_path", "/tmp"))
    path_provider = YMDPathProvider(
        filename_provider=UUIDFilenameProvider(),
        base_directory_path=path,
        create_dir_depth=-4,
    )
    return path_provider

# File: sim_detector.py
from ophyd_async.core import PathProvider
from ophyd_async.epics.adsimdetector import SimDetector as SimDetectorBase

from .area_detectors import default_path_provider

class SimDetector(SimDetectorBase):
    def __init__(self, prefix, path_provider: PathProvider | None = None, **kwargs):
        if path_provider is None:
            path_provider = default_path_provider()
        super().__init__(prefix=prefix, path_provider=path_provider, **kwargs)

This produces area detectors files like:

area_detector_files/
├─vortex_me4/
│ └─2026/
│   └─02/
│     ├─19/
│     │ ├─40f23255-aa07-4231-afcf-00f589e1c353.h5
│     │ └─ecd2e6f7-b509-4fc3-8654-d5f84a7ef6d8.h5
│     └─20/
│       └─1580638a-75e4-4895-867b-0828845d33e3.h5
└─ge_13element/
  └─2026/
    └─02/
      ├─19/
      │ └─40f23255-aa07-4231-afcf-00f589e1c353.h5
      └─20/
        └ 41301fb5-3df2-49c7-b978-ede5cd1ea069.h5

Option 2: EPICS

We could add a PV that would hold the current DM experiment
name. Then the path provider would read this PV and incorporate
that experiment name into the path.

Pros:

• Easy to set, just use a pydm or caQtDM widget

Cons:

• Hard to read
  • Ophyd can read the PV maybe with direct method access (Signal.get())?
  • Can't use the path provider protocol since it's not asynchronous
    • Maybe we override each device's prepare() method?
• Adds beamline-global state
  • Ideally, the DM Experiment would only be a metadata key in the start doc
  • What happens if a second python REPL changes the DM experiment?
• Two places to set DM experiment: EPICS PV and start document
• What should we do for scans without a DM experiment?

Option 3: Bluesky Pre-processors

We could write a Bluesky pre-processor that looks for the DM
experiment name in the start document. Then it could set this as an
attribute on the ophyd path provider in real time. Something like:

# File: dm_path_provider.py
class DMPathProvider(PathProvider):
    dm_experiment: str | None = None

    def __call__(self, device_name: str | None = None) -> PathInfo:
        filename = self._filename_provider(device_name)
        directory_uri = self.dm_experiment || "no_experiment"
        return PathInfo(
            directory_uri=directory_uri,
            filename=filename,
            create_dir_depth=self._create_dir_depth,
        )

# File: add_dm_path.py
def dm_experiment_preprocessor(plan, path_provider):
    …
    # Wait for a start document, and if we find one then update the path provider
    if (dm_exp := md.get('dm_experiment')):
        path_provider.dm_experiment = dm_exp

# File: startup.py
path_provider = DMPathProvider()
callback = partial(dm_experiment_preprocessor, path_provider=path_provider)
RE.subscribe(callback)

detector = EigerDetector(…, path_provider=path_provider)

Pros:

• Gets us DM access control in the filesystem
• No global state
• Compatible with existing ophyd-async detector classes

Cons:

• Complex
• Devices need to share a path provider or each device needs its own callback
• What should we do for scans without a DM experiment?

Option 4: A Bluesky Plan

We could add a bluesky plan experiment_setup() that accepts a DM experiment and updates the path provider. Would probably need to be curried, i.e. experiment_setup = build_experiment_setup(path_provider).

Then we can use the bps.wait_for() plan-stub to do a network I/O to get the experiment's data directory.

Pros:

• Explicit
• No beamline-global state (no PVs)

Cons:

• Complicated
• Devices need to share a path provider or each device needs its own callback
• Queueserver plan order becomes critical

[prjemian]

The plan can receive the experiment name, either directly as a plan (kw)arg or indirectly (such as Python global, class attribute, or EPICS PV). DM will provide the dataDirectory based on the experiment name. The plan, per its design, can configure its various controls and file writers to use appropriate subdirectories.

[prjemian]

import pathlib

from dm import DsApiFactory

# Make sure DM's environment variables are defined before the next command.
dsapi = DsApiFactory.getExperimentDsApi()

# ... later, in the plan
    dm_data_dir = pathlib.Path(dsapi.getExperimentByName(experiment_name)["dataDirectory"])

Edit: removed unneeded call of dsapi().

[prjemian]

Other comments in private Teams discussion.

[prjemian]

Both Bluesky runs and APS Data Management experiments have their own metadata. It is a good idea to copy certain keys from one to the other. Here is a minimum:

• The APS DM experiment name is good to add to a bluesky run.
• The bluesky run uid is good to add to the DM experiment. (Accumulate in a list, an experiment may have many bluesky runs.)

[canismarko]

The bluesky run uid is good to add to the DM experiment. (Accumulate in a list, an experiment may have many bluesky runs.)

This would probably also be a preprocessor, right?

[prjemian]

I usually add this to DM metadata at the time I start the DM workflow from within my bluesky plan. I believe that is the first time to add such metadata. It links the specific DM workflow run (identified by its own uuid) with the bluesky run uid.

How to do this when there is no DM workflow? Not sure. Perhaps deposit in a metadata file in the run's working directory.