Add MID-XFEL loader, geometry and pipeline integration#111
Draft
Abd-zak wants to merge 10 commits into
Draft
Conversation
Abd-zak
changed the title
added 9 commits
June 1, 2026 12:52
Add European XFEL loading support based on DAMNIT processed variables and EXtra-data aliases. This updates XFELLoader to: - load detector images from DAMNIT variables - reduce pulse-resolved data using mean, sum, or no reduction - replace NaN/inf detector values with zeros before downstream processing - load run metadata through EXtra-data RunDirectory and alias files - read rocking-angle positions from the DAMNIT xarray position coordinate - load detector distance from the sdd alias - load photon energy from the energy-kev alias and convert keV to eV - infer the scanned motor when no explicit angle array is provided - keep a fallback scan-position detector for train-resolved motor values The implementation was validated against the MID notebook reference for: - detector data shape and values - rocking-angle positions - detector distance - photon energy.
The XFEL loader was simplified by removing the dedicated
`damnit_dir_name` configuration and using the provided
`experiment_file_path` directly as the DAMNIT database root.
Changes:
- remove `damnit_dir_name` parameter from XFELLoader
- remove associated instance attribute and path construction logic
- simplify DAMNIT database access by using:
Damnit(experiment_file_path)
- move DAMNIT import to module level alongside EXtra-data imports
- simplify loader initialization and configuration interface
- update safe-loader documentation strings
- preserve existing EXtra-data RunDirectory support
- preserve alias-file handling through `extra-data-aliases.yml`
- preserve pulse reduction workflow (mean/sum/None)
- preserve rocking-angle detection and geometry handling
- preserve detector calibration, energy loading, and motor loading APIs
- preserve NaN/Inf sanitization of detector data through
`np.nan_to_num`
Motivation:
The previous implementation required a separate DAMNIT
subdirectory configuration although the loader already operates
from a DAMNIT database location. Removing this extra parameter
reduces configuration complexity and makes the XFEL loader API
closer to the actual XFEL workflow.
Notes:
- detector images continue to be loaded from DAMNIT variables
- NaN values present in raw detector data are still converted
to zeros before downstream processing
- geometry mapping verification remains pending
- testing on reconstructed datasets will be performed separately
- move DAMNIT and EXtra-data from optional dependencies to core dependencies - remove the dedicated xfel extra - simplify dependency management for XFEL support - ensure XFEL packages are available in standard cdiutils installations - avoid import failures when XFEL modules are imported - remove duplicate XFEL dependency definitions from optional extras
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces an initial implementation of support for BCDI datasets acquired at the European XFEL MID instrument.
The implementation follows the existing cdiutils beamline architecture and provides a new XFEL loader together with the required geometry and pipeline integration to enable data loading and preprocessing through the standard cdiutils workflow.
Changes
New XFEL loader
Added:
providing:
LoaderinterfaceThe implementation is currently based on MID proposal
p010402and uses the EXtra-data / DAMNIT ecosystem for accessing processed experimental data.Loader registration
Registered XFEL support in:
allowing:
and:
Geometry support
Added XFEL geometry definition in:
using the geometry currently employed in the XFEL reciprocal-space mapping workflow.
The geometry includes:
allowing direct integration with
SpaceConverterand the standard cdiutils preprocessing pipeline.AGIPD detector support
Added AGIPD detector support in:
including detector recognition and mask generation through the existing cdiutils infrastructure.
Detector calibration parameters are automatically loaded, including:
NaN handling
The processed detector data provided through DAMNIT may contain NaN values.
To ensure compatibility with the preprocessing pipeline and voxel-centering routines, NaN values are automatically replaced by zero during detector data loading.
Example notebook
Added:
demonstrating:
Validation
The loader outputs were compared directly against the reference XFEL analysis notebook.
Detector data
Motor positions
Experimental parameters
The following quantities were verified against the reference workflow:
All values were found to be identical.
Current status
The implementation has been validated for:
Validation on reconstructed BCDI datasets and full reconstruction workflows is currently ongoing and will be performed on upcoming XFEL experimental data.