Skip to content

Add detailed step reference docs #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lg345 merged 1 commit into from
Jul 17, 2026
Merged

Add detailed step reference docs #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 127 additions & 0 deletions docs/steps/binning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Axes, bin indices & binned reductions

Steps that build delay/energy axes, assign each shot to a bin, and collapse
per-shot data into a binned spectrum. Axis and index steps run first; the
`reduce_detector_*` steps consume the indices they produce. See the
[step reference overview](index.md) for the shape vocabulary.

## Building axes and bin indices

### `time_binning`
Compute per-shot delays from laser timing keys and lay down delay bins.

- **Reads:** `lxt_key`, `fast_delay_key`, `tt_correction_key` — each `(shots,)`.
Uses whichever exist.
- **Writes:** `delays` `(shots,)`, `time_bins`, `time_bins_centered`,
`timing_bin_indices` `(shots,)`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `bins` | required | `"auto"`, an explicit list of centers, or `[min, max, num_points]` |
| `lxt_key` | `"lxt_ttc"` | long-delay stage key; `null` to skip |
| `fast_delay_key` | `"encoder"` | fast-stage key |
| `tt_correction_key` | `"time_tool_correction"` | time-tool jitter correction key |
| `resolution` | `50e-15` | bin width in seconds for `bins: auto` |

### `make_ccm_axis`
Build incident-energy (CCM) bin edges and centers for an XAS scan.

- **Reads:** `ccm_key` `(shots,)` when `energies: auto`.
- **Writes:** `ccm_bins` — `n+1` edges; `ccm_energies` — `n` centers.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `energies` | `"auto"` | `"auto"`, explicit list, or `[min, max, num_points]` |
| `ccm_key` | `"ccm"` | per-shot incident-energy key |
| `resolution` | `0.001` | bin width (keV) for `energies: auto` |

### `ccm_binning`
Assign each shot to a CCM energy bin.

- **Reads:** `ccm_key` `(shots,)`, `ccm_bins_key` (edges).
- **Writes:** `ccm_bin_indices` `(shots,)`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `ccm_key` | `"ccm"` | per-shot incident-energy key |
| `ccm_bins_key` | `"ccm_bins"` | edges produced by `make_ccm_axis` |

### `bin_uniques`
Bin an arbitrary scan variable by its unique values (one bin per value).

- **Reads:** scan variable `on` `(shots,)`.
- **Writes:** `scanvar_indices` `(shots,)`, `scanvar_bins` (unique values).
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | per-shot scan variable to bin |

### `make_energy_axis`
Convert pixel index to emission energy from von Hamos crystal geometry.

- **Reads:** `detector_key` (to read pixel count) or `n_pixels` directly.
- **Writes:** `<name>_energy` `(pixels,)`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `crystal_detector_distance` | required | crystal-to-detector distance A (mm) |
| `crystal_radius` | required | crystal bend radius R (mm) |
| `d_spacing` | required | crystal d-spacing (angstrom) |
| `detector_key` | `None` | key to read pixel count from |
| `n_pixels` | `None` | pixel count; overrides `detector_key` |
| `mm_per_pixel` | `0.05` | pixel pitch (mm) |
| `name` | `"xes"` | output prefix, so the axis is `<name>_energy` |

## Binned reductions

Each reduction sums per-shot data into its bins. Pass `average: True` to divide
by the per-bin count; the raw count is always written as `<on>_..._bincount` for
downstream normalization or cross-run combination.

### `reduce_detector_temporal`
Bin a per-shot spectrum along delay.

- **Reads:** detector `on` — 1D `(shots,)` or 2D `(shots, pixels)`; plus
`timing_bin_indices` and `time_bins`.
- **Writes:** `<on>_time_binned` `(n_time_bins[, pixels])`, `<on>_bincount`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `timing_bin_key` | `"timing_bin_indices"` | bin-index attribute |
| `average` | `False` | divide each bin by its shot count |

### `reduce_detector_ccm`
Bin a per-shot spectrum along incident energy.

- **Reads:** detector `on` — 1D/2D/3D; plus `ccm_bin_indices`.
- **Writes:** `<on>_energy_binned` `(n_energy[, ...])`, `<on>_energy_bincount`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `ccm_bin_key` | `"ccm_bin_indices"` | bin-index attribute |
| `average` | `False` | divide each bin by its shot count |

### `reduce_detector_ccm_temporal`
Bin a per-shot spectrum along both delay and incident energy (2D map).

- **Reads:** detector `on` — 1D/2D; plus `timing_bin_indices` and
`ccm_bin_indices`.
- **Writes:** `<on>_time_energy_binned` `(n_time, n_energy[, pixels])`,
`<on>_time_energy_bincount`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `timing_bin_key` | `"timing_bin_indices"` | delay bin-index attribute |
| `ccm_bin_key` | `"ccm_bin_indices"` | energy bin-index attribute |
| `average` | `False` | divide each bin by its shot count |
145 changes: 145 additions & 0 deletions docs/steps/detector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Detector correction, geometry & spatial reduction

Steps that clean up detector frames, straighten a tilted dispersion axis, and
collapse the spatial axes into a spectrum. See the
[step reference overview](index.md) for the shape vocabulary and the read/write
model.

## Correction

### `common_mode_correction`
Subtract a per-row, per-column, or per-bank baseline estimated from a
signal-free band. Shape preserved.

- **Reads:** detector `on` — 3D `(shots, rows, cols)` or 2D `(rows, cols)`.
- **Writes:** overwrites `on`; same shape.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `axis` | `"row"` | `row` (per-row offset across columns), `column`, or `bank` |
| `method` | `"median"` | `median` (robust) or `mean` |
| `reference` | full extent | `[start, end]` of the dark band, indexed on the axis orthogonal to `axis` |
| `bank_size` | `128` | column width of an ePix100 bank; only used when `axis: bank` |

The `reference` range is a column range for `axis: row` and a row range for
`axis: column` or `axis: bank`.

```yaml
- step: common_mode_correction
on: epix
axis: row
reference: [0, 40] # dark columns
```

### `patch_pixels`
Repair bad pixels or columns, either from an explicit list or auto-detected.

- **Reads:** detector `on` — 1D, 2D, or 3D.
- **Writes:** overwrites `on`; same shape.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `pixels` | `None` | explicit list of indices/columns to patch |
| `auto_detect` | `False` | find bad columns from the data instead of a list |
| `threshold` | `5.0` | auto-detect: deviation cut for flagging a column |
| `nsigma` | `5.0` | auto-detect: sigma multiplier for the flag |
| `max_gap_width` | `4` | widest run of bad pixels to bridge |
| `smooth_window` | `31` | window for the smoothed reference profile |
| `mode` | `"polynomial"` | `polynomial`, `interpolate`, or `zero` |
| `axis` | last | axis to patch along |
| `patch_range` | `4` | half-width of the neighborhood sampled around a bad pixel |
| `poly_range` | `6` | half-width of the fit window |
| `deg` | `1` | polynomial degree for `mode: polynomial` |

## Geometry

### `find_rotation_angle`
Auto-detect the tilt of a dispersed signal so `rotate_detector` can straighten
it.

- **Reads:** detector `on` — 3D `(shots, rows, cols)` or 2D.
- **Writes:** `<on>_angle` (override with `angle_key`) — float degrees.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `low_threshold` | `30` | lower ADU bound for edge detection |
| `high_threshold` | `100` | upper ADU bound for edge detection |
| `angle_key` | `<on>_angle` | attribute to store the detected angle |

### `rotate_detector`
Rotate detector frames by a fixed angle or by a previously detected one.

- **Reads:** detector `on` — 3D or 2D.
- **Writes:** overwrites `on`.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `angle` | `0` | rotation in degrees |
| `angle_key` | `None` | read the angle from this attribute; takes precedence over `angle` |
| `axes` | `[1, 2]` 3D / `[0, 1]` 2D | plane to rotate in |
| `reshape` | `False` | grow the output to fit the rotated frame vs keep shape |

```yaml
- step: find_rotation_angle
on: epix
angle_key: epix_angle
- step: rotate_detector
on: epix
angle_key: epix_angle
```

## Spatial reduction

### `apply_roi`
Crop to one or more regions of interest, keeping the spatial dimension.

- **Reads:** detector `on` — 2D or 3D.
- **Writes:** `<on>_ROI_n` (one per ROI, or combined) — spatial axes retained.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `rois` | required | list of `[row0, row1, col0, col1]` crops |
| `combine_rois` | `True` | merge ROIs into one array vs keep separate `_ROI_n` |

### `reduce_detector_spatial`
Crop to ROIs and collapse one spatial axis into a per-shot dispersion trace.

- **Reads:** detector `on` — 3D `(shots, rows, cols)` or 2D.
- **Writes:** `<on>_ROI_n` — reduced along the chosen axis.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `rois` | required | list of `[row0, row1, col0, col1]` crops |
| `combine_rois` | `True` | merge ROIs vs keep separate |
| `reduction` | `"sum"` | `sum` or `mean` over the reduced axis |
| `axis` | `1` (3D) / `-1` | spatial axis to collapse |
| `purge` | `True` | drop the source key after reducing to free memory |

ROI rows are given in absolute detector coordinates; the step translates them
into the loaded crop using the run's `_row_offset`, so the same YAML works
whether the detector was cropped at load or not.

### `reduce_detector_shots`
Collapse the shot axis into a single averaged or summed frame.

- **Reads:** detector `on` — shot axis is axis 0.
- **Writes:** `<on>_reduced` — shot axis removed.
- **Parameters:**

| name | default | description |
|------|---------|-------------|
| `on` | required | detector key |
| `reduction` | `"sum"` | `sum` or `mean` over shots |
| `purge` | `True` | drop the source key after reducing |
93 changes: 93 additions & 0 deletions docs/steps/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Step reference

Every registered pipeline step, grouped by what it does, with the arrays it
reads, the arrays it writes, and the parameters you set in YAML.

For the auto-generated, source-linked docstrings see
[Analysis steps (API)](../source_analysis.md). For how steps fit into a full
pipeline file, see the [YAML pipeline guide](../YAML_PIPELINE_GUIDE.md).

## The step contract

A step is a stateless function `step(run, **kwargs) -> None`. The `run` object
is a shared blackboard: a step reads attributes off it, does its work, and
writes results back as new attributes. The next step picks those up by name.
Nothing is returned; everything flows through `run`.

So in this reference:

- **Reads** = the run attributes (and their shapes) a step expects to already exist.
- **Writes** = the run attributes it sets.
- **Parameters** = the keys you pass in the YAML `step:` block. `on` is almost
always the attribute a step operates on.

A step that can't find its input (`on` missing, or the attribute is `None`)
returns quietly and logs a status message rather than raising. This lets the
same pipeline run on a parent that has no detector loaded (batched path) and on
each batch that does.
Comment on lines +24 to +27

## Array shapes

The docs use a consistent vocabulary for array shapes:

| Shape | Meaning |
|-------|---------|
| `(shots, rows, cols)` | 3D detector, one frame per shot |
| `(shots, pixels)` | 2D detector after one spatial axis is reduced |
| `(shots,)` | per-shot scalar or 1D key (IPM, delay, CCM energy) |
| `(shots,)` bool | shot mask (`xray`, `laser`, `simultaneous`) |
| `(n_time_bins, pixels)` or `(n_time_bins,)` | time-binned spectrum |
| `(n_energy, pixels)` or `(n_energy,)` | energy (CCM) binned spectrum |
| `(n_time, n_energy[, pixels])` | 2D time+energy binned |

Detector steps generally accept 2D or 3D and act on the shot axis (axis 0) or
the last (dispersion) axis; each entry says which.

## Naming conventions

Steps build output names from the input key plus a suffix describing the
operation, so a chain reads left to right:

```
epix (loaded detector, 3D)
epix_ROI_1 (reduce_detector_spatial)
epix_ROI_1_simultaneous_laser (union_shots)
epix_ROI_1_simultaneous_laser_time_binned (reduce_detector_temporal)
epix_ROI_1_simultaneous_laser_time_binned_normalized (normalize_xes)
```

Each step's `on` is the previous step's output name.

## All steps

| Step | Group | Reads | Writes |
|------|-------|-------|--------|
| [`load_run_keys`](loading_filtering.md#load_run_keys) | Loading | HDF5 | named per-shot keys |
| [`load_detector`](loading_filtering.md#load_detector) | Loading | HDF5 | 3D detector |
| [`get_run_shot_properties`](loading_filtering.md#get_run_shot_properties) | Loading | lightStatus | `xray`, `laser`, `simultaneous` |
| [`droplet_reconstruction`](loading_filtering.md#droplet_reconstruction) | Loading | sparse HDF5 | `new_key` 3D stack |
| [`filter_shots`](loading_filtering.md#filter_shots) | Filtering | mask + key | overwrites mask |
| [`filter_detector_adu`](loading_filtering.md#filter_detector_adu) | Filtering | detector | overwrites `on` |
| [`filter_detector_variance`](loading_filtering.md#filter_detector_variance) | Filtering | 3D detector | overwrites `on` + `_variance_mask` |
| [`hitfinding`](loading_filtering.md#hitfinding) | Filtering | 3D detector | overwrites `on` (fewer shots) |
| [`union_shots`](loading_filtering.md#union_shots) | Filtering | `on` + masks | `new_key` |
| [`separate_shots`](loading_filtering.md#separate_shots) | Filtering | `on` + 2 masks | `new_key` |
| [`common_mode_correction`](detector.md#common_mode_correction) | Detector | 3D/2D detector | overwrites `on` |
| [`patch_pixels`](detector.md#patch_pixels) | Detector | detector | overwrites `on` |
| [`find_rotation_angle`](detector.md#find_rotation_angle) | Detector | 3D/2D detector | `<on>_angle` |
| [`rotate_detector`](detector.md#rotate_detector) | Detector | 3D/2D detector | overwrites `on` |
| [`apply_roi`](detector.md#apply_roi) | Spatial | 2D/3D detector | `<on>_ROI_n` (keeps spatial) |
| [`reduce_detector_spatial`](detector.md#reduce_detector_spatial) | Spatial | 3D/2D detector | `<on>_ROI_n` (reduced) |
| [`reduce_detector_shots`](detector.md#reduce_detector_shots) | Spatial | detector | `<on>_reduced` |
| [`time_binning`](binning.md#time_binning) | Binning | timing keys | `time_bins`, `timing_bin_indices` |
| [`make_ccm_axis`](binning.md#make_ccm_axis) | Binning | `ccm` | `ccm_bins`, `ccm_energies` |
| [`ccm_binning`](binning.md#ccm_binning) | Binning | `ccm`, `ccm_bins` | `ccm_bin_indices` |
| [`bin_uniques`](binning.md#bin_uniques) | Binning | scan var | `scanvar_indices`, `scanvar_bins` |
| [`make_energy_axis`](binning.md#make_energy_axis) | Binning | detector shape | `<name>_energy` |
| [`reduce_detector_temporal`](binning.md#reduce_detector_temporal) | Binning | 1D/2D + time indices | `<on>_time_binned` |
| [`reduce_detector_ccm`](binning.md#reduce_detector_ccm) | Binning | 1D/2D/3D + ccm indices | `<on>_energy_binned` |
| [`reduce_detector_ccm_temporal`](binning.md#reduce_detector_ccm_temporal) | Binning | 1D/2D + both indices | `<on>_time_energy_binned` |
| [`normalize_xes`](spectra.md#normalize_xes) | Spectra | 1D/2D spectrum | `<on>_normalized` |
| [`subtract_polynomial_background`](spectra.md#subtract_polynomial_background) | Spectra | 1D/2D spectrum | `<on>_bkgsub` |
| [`purge_keys`](spectra.md#purge_keys) | Utility | nothing | sets keys to `None` |
| [`combine_runs`](spectra.md#combine_runs) | Reduction | per-run binned data | results dict |
Loading