mfx102101026 Fe Kα pipeline steps: event-code mask, spatial background, smalldata override, Zr calibration - #107
Merged
Merged
Conversation
Allow the YAML experiment section to specify an explicit smalldata_dir,
overriding the default /sdf/data/lcls/ds/{hutch}/{exp}/hdf5/smalldata
location search. Enables running the pipeline on locally reprocessed
smalldata (e.g. custom pedestal/droplet params) without touching the
shared experiment directory.
Estimate a per-line fluorescence background from side-band regions flanking a dispersed spectral streak (on the cross-dispersion axis) and subtract it from the signal band. Removes the isotropic fluorescence pedestal under a Von Hamos emission line while preserving the lineshape, before spatial reduction to a 1D spectrum.
Extract one EVR/timing event-code column as a per-shot boolean mask so it can be consumed by union_shots/filter_shots. Needed to gate 30 Hz beam delivery (event code 198) inside a 120 Hz DAQ, which the smalldata xray flag does not distinguish. Used by the mfx101592326 Fe Ka droplet pipeline.
There was a problem hiding this comment.
Pull request overview
This PR brings several mfx102101026 / Fe Kα XES pipeline enhancements onto master, adding new configuration options and analysis steps for shot gating, spatial background subtraction, and energy-axis calibration using known foil emission lines.
Changes:
- Add
smalldata_diroverride support (config parsing + experiment directory resolution + pipeline wiring). - Introduce new registered analysis steps:
make_eventcode_mask,subtract_spatial_background, andcalibrate_energy(+ emission line table). - Update the mfx102101026 aggregate notebook (but currently with committed execution outputs/errors).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| XSpect/model/experiment.py | Adds smalldata_dir override handling in experiment directory lookup. |
| XSpect/controller/pipeline.py | Passes smalldata_dir from parsed config into spectroscopy_experiment. |
| XSpect/controller/config_parser.py | Extends YAML schema to include optional experiment.smalldata_dir. |
| XSpect/analysis/xes.py | Adds emission-line table and new calibrate_energy step. |
| XSpect/analysis/spectroscopy.py | Adds new steps for event-code masking and spatial background subtraction. |
| experiments/mfx102101026/mfx102101026_aggregate_xes.ipynb | Notebook updated, but includes committed outputs (stdout + traceback). |
Comments suppressed due to low confidence (1)
experiments/mfx102101026/mfx102101026_aggregate_xes.ipynb:82
- This notebook cell has committed execution state and an error traceback in outputs. Committing failing outputs makes the notebook look broken and creates large, unstable diffs. Clear outputs and reset
execution_countbefore committing.
"execution_count": 2,
"id": "cell-load",
"metadata": {},
"outputs": [
{
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+48
to
+54
| if self.smalldata_dir: | ||
| if os.path.exists(self.smalldata_dir) and os.listdir(self.smalldata_dir): | ||
| self.experiment_directory = self.smalldata_dir | ||
| return self.smalldata_dir | ||
| raise Exception( | ||
| f"smalldata_dir '{self.smalldata_dir}' does not exist or is empty." | ||
| ) |
Comment on lines
290
to
291
| except Exception: | ||
| exp = _MockExperiment(cfg.lcls_run, cfg.hutch, cfg.experiment_id) |
Comment on lines
+75
to
+92
| code = kwargs.get("code") | ||
| ec_key = kwargs.get("eventcodes", "eventcodes") | ||
| new_key = kwargs.get("new_key", None) | ||
| if code is None: | ||
| run.update_status("make_eventcode_mask: 'code' is required") | ||
| return | ||
| table = getattr(run, ec_key, None) | ||
| if table is None: | ||
| run.update_status( | ||
| f"make_eventcode_mask: '{ec_key}' not loaded (add it to data.keys)" | ||
| ) | ||
| return | ||
| table = np.asarray(table) | ||
| if table.ndim != 2 or code >= table.shape[1]: | ||
| run.update_status( | ||
| f"make_eventcode_mask: '{ec_key}' shape {table.shape} cannot index code {code}" | ||
| ) | ||
| return |
Comment on lines
+574
to
+587
| # Move the cross-dispersion axis to position 1 so we can index columns. | ||
| work = data if bkg_axis == 1 else data.T # (lines, cross) | ||
| est_fn = np.nanmedian if estimator == "median" else np.nanmean | ||
|
|
||
| # Per-line background level (one value per dispersion line) from side bands. | ||
| side_pixels = [] | ||
| for lo, hi in sidebands: | ||
| side_pixels.append(work[:, lo:hi]) | ||
| side = np.concatenate(side_pixels, axis=1) | ||
| bkg_per_pixel = est_fn(side, axis=1, keepdims=True) # (lines, 1) | ||
|
|
||
| out = work.astype(float).copy() | ||
| s0, s1 = signal | ||
| out[:, s0:s1] = out[:, s0:s1] - bkg_per_pixel # subtract per-pixel bkg |
Comment on lines
+178
to
+180
| if spec_key is None or R is None or d is None: | ||
| run.update_status("calibrate_energy: 'on', crystal_radius, d_spacing required") | ||
| return |
Comment on lines
+206
to
+213
| n_pixels = kwargs.get("n_pixels", spec.shape[-1]) | ||
|
|
||
| # Locate the peak pixel (optionally restricted to a window) | ||
| if peak_window is not None: | ||
| lo, hi = int(peak_window[0]), int(peak_window[1]) | ||
| p0 = lo + int(np.argmax(spec[lo:hi])) | ||
| else: | ||
| p0 = int(np.argmax(spec)) |
Comment on lines
+31
to
+33
| smalldata_dir: "str | None" = None # Optional override for the smalldata | ||
| # directory. If set, XSpect looks for {exp}_Run{run:04d}.h5 here instead of | ||
| # the default /sdf/data/lcls/ds/{hutch}/{exp}/hdf5/smalldata search list. |
Comment on lines
+138
to
+140
| @register_step("calibrate_energy") | ||
| def calibrate_energy(run, **kwargs): | ||
| """Calibrate the vonHamos crystal_detector_distance (A) against a known line. |
Comment on lines
+57
to
+60
| @register_step("make_eventcode_mask") | ||
| def make_eventcode_mask(run, **kwargs): | ||
| """Build a per-shot boolean mask from an EVR/timing event-code column. | ||
|
|
Comment on lines
+29
to
+33
| "execution_count": 1, | ||
| "id": "cell-setup", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "outputs": [ | ||
| { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the four pipeline features developed for the mfx102101026 / mfx101592326 Fe Kα XES analysis onto
master:make_eventcode_mask— extract one EVR/timing event-code column as a per-shot boolean mask (gates 30 Hz beam delivery on code 198 inside a 120 Hz DAQ, which the smalldataxrayflag does not distinguish).subtract_spatial_background— per-row background subtraction from flanking sideband columns (supports asymmetric/one-sided sidebands, e.g. to avoid tape-contaminated columns).Notes
masteris ~19 commits ahead (docs +common_mode_correction/subtract_polynomial_background); a merge may need conflict resolution inspectroscopy.pywhere both sides register new steps.