Architecture overhaul: YAML-driven pipeline with MVC separation

[lg345]

Summary

Overhaul XSpect to use YAML-based configuration files as the single source of truth for analysis pipelines. Replace the current controller subclass hierarchy (6 classes with hardcoded analysis sequences) with a generic pipeline runner that dispatches steps from a YAML recipe.

Motivation

• The controller has become a graveyard for bespoke analysis sequences. Every new experiment type requires a new subclass.
• Configuration is scattered across object attributes set imperatively in notebooks.
• No reproducibility artifact exists. You can't hand someone a file and say "this is exactly what I ran."
• The controller does analysis work instead of mediating it.

Architecture

YAML as the complete recipe

One YAML file captures everything needed to reproduce an analysis from raw data:

description: "Mn Kbeta XES, time-resolved pump-probe, MFX LY69"

experiment:
  hutch: xcs
  experiment_id: xcsp23820
  lcls_run: 21

data:
  runs: [162-181, 183-190]
  keys:
    tt/ttCorr: time_tool_correction
    epics/lxt_ttc: lxt_ttc
    enc/lasDelay: encoder
    ipm4/sum: ipm
    tt/AMPL: time_tool_ampl
  detector_keys:
    epix_1/ROI_0_area:
      name: epix
      rois: [[0, -1]]
      combine_rois: true

pipeline:
  - step: filter_shots
    on: xray
    filter_key: ipm
    threshold: 1.0e4
  - step: union_shots
    on: epix_ROI_1
    filters: [simultaneous, laser]
  - step: separate_shots
    on: epix_ROI_1
    filters: [xray, laser]
  - step: reduce_detector_temporal
    on: epix_ROI_1.simultaneous_laser
    timing: timing_bin_indices.simultaneous_laser

reduction:
  - step: combine_runs
  - step: normalize_xes
    on: summed_laser_off

output:
  format: hdf5
  path: ./results/

Target module layout

XSpect/
├── __init__.py
├── model/
│   ├── experiment.py        # experiment, spectroscopy_experiment
│   ├── run.py               # spectroscopy_run (results dict interface)
│   └── von_hamos.py         # vonHamos crystal geometry
├── analysis/
│   ├── registry.py          # @register_step, @register_reduction, dispatch
│   ├── spectroscopy.py      # base operations (filter, union, separate, reduce)
│   ├── xes.py               # XES-specific steps (normalize, energy axis, combine)
│   └── xas.py               # XAS-specific steps (ccm axis, ccm binning)
├── controller/
│   ├── config_parser.py     # YAML parsing + validation
│   ├── pipeline_runner.py   # step dispatch loop + reduction orchestration
│   └── batch_manager.py     # shot chunking, multiprocessing, reconvergence
├── visualization/
├── diagnostics/
└── postprocessing/

---

Phase Structure (with dependencies)

Phase 1: MVP (Static XES end-to-end)

1A: Core Infrastructure (#84)

Dependencies: None (foundational)

Deliverable	Description
model/run.py	spectroscopy_run refactored with self.results = {} flat dict
model/experiment.py	Move experiment, spectroscopy_experiment as-is
analysis/registry.py	@register_step(name), @register_reduction(name), get_step(), get_reduction(), list_steps()
controller/config_parser.py	YAML parse, validate required sections (experiment, data, pipeline, output), raise on unknown step names
controller/pipeline_runner.py	Iterate pipeline section, look up + call steps, log execution
controller/batch_manager.py	Split runs into shot ranges, multiprocessing.Pool, reconverge batch results
Pipeline class	Pipeline.from_yaml(path), .run(cores, batch_size), .results

Acceptance criteria:

• Pipeline.from_yaml("test.yaml").run(cores=4, batch_size=1000) executes on a minimal YAML with placeholder steps
• Registry discovers and dispatches decorated methods
• Batch manager splits and reconverges correctly

---

1B: Register Static XES Steps (#85)

Dependencies: 1A (registry must exist before steps can be registered)

Step	Source	Description
filter_shots	SpectroscopyAnalysis.filter_shots	Threshold filter on diagnostic key
union_shots	SpectroscopyAnalysis.union_shots	Combine masks (AND logic)
separate_shots	SpectroscopyAnalysis.separate_shots	Exclude masks (A AND NOT B)
filter_detector_adu	SpectroscopyAnalysis.filter_detector_adu	Zero pixels below ADU threshold
reduce_detector_shots	SpectroscopyAnalysis.reduce_detector_shots	Sum detector across shot dimension
reduce_detector_spatial	SpectroscopyAnalysis.reduce_detector_spatial	ROI reduction of spatial dimension
apply_roi	SpectroscopyAnalysis.apply_roi	Apply ROI mask
rotate	wraps scipy.ndimage.rotate	Rotate detector images
patch_pixels	SpectroscopyAnalysis.patch_pixels	Bad pixel interpolation
hit_finding	HitFinding.basic_detect	Event detection + filter

Key changes:

• All steps become stateless: no self.pixels_to_patch, no self.adu_cutoff. Parameters from YAML args only.
• Steps read/write run.results[key] instead of setattr(run, key, val).
• Dot-separated key convention: union_shots on epix_ROI_1 with filters [simultaneous, laser] writes to run.results["epix_ROI_1.simultaneous_laser"].

Acceptance criteria:

• Each step callable independently with mock run + YAML-style kwargs
• Correct output keys following dot-separated naming
• Status logs written for every step execution

---

1C: Static XES Integration (#86)

Dependencies: 1A + 1B (all steps registered, pipeline infrastructure runnable)

Deliverable	Description
xcsp23820_static.yaml	Complete config reproducing XSpect_P23820.py static workflow
model/von_hamos.py	Move vonHamos class to model layer
Numerical validation	Run old path + new path on same data, compare outputs
combine_runs reduction	Basic cross-run summation (reduction lifecycle test)

Acceptance criteria:

• Pipeline.from_yaml("xcsp23820_static.yaml").run(cores=16, batch_size=2000) produces results matching old XESBatchAnalysisRotation.primary_analysis_static
• Execution log captures all step statuses
• Results saved to HDF5 at specified path

---

Phase 2: Time-Resolved XES (#87)

Dependencies: Phase 1 complete (1C passes numerical validation)

New pipeline steps

Step	Source	Description
time_binning	SpectroscopyAnalysis.time_binning	Bin shots by delay stage + encoder + time tool
reduce_detector_temporal	SpectroscopyAnalysis.reduce_detector_temporal	Bin detector data into time bins
normalize_xes	XESAnalysis.normalize_xes	Area-normalize XES spectra
make_energy_axis	XESAnalysis.make_energy_axis	Energy axis from von Hamos geometry
droplet_reconstruction	SpectroscopyAnalysis.droplet_reconstruction	Sparse-to-dense detector reconstruction

New reduction steps

Step	Source	Description
combine_runs (full)	XESAnalysis.combine_runs	Sum across runs with uncertainty propagation
normalize_combined	Extracted from combine_runs	Normalize combined laser-on/off spectra
compute_difference	Extracted from combine_runs	Normalized difference spectra

Acceptance criteria:

• Full time-resolved XES pipeline runs from YAML
• Results match current XESBatchAnalysisRotation.primary_analysis output
• Energy axis correctly generated from von Hamos parameters

---

Phase 3: XAS Pipelines (#88)

Dependencies: Phase 2 complete (time_binning shared between time-resolved XES and XAS)

New pipeline steps

Step	Source	Description
make_ccm_axis	XASAnalysis.make_ccm_axis	CCM energy bins from setpoint values
ccm_binning	XASAnalysis.ccm_binning	Digitize shots into CCM energy bins
reduce_detector_ccm	XASAnalysis.reduce_detector_ccm	1D XAS (energy only)
reduce_detector_ccm_temporal	XASAnalysis.reduce_detector_ccm_temporal	2D XAS (energy + time)
bin_uniques	SpectroscopyAnalysis.bin_uniques	Bin by arbitrary scan variable

Workflows replaced

Old class	New YAML
XASBatchAnalysis	2D (energy + time) YAML
XASBatchAnalysis_1D_ccm	Energy-only YAML
XASBatchAnalysis_1D_time	Time-only YAML

Acceptance criteria:

• All three XAS workflows (1D energy, 1D time, 2D energy+time) run from YAML
• Results match current XAS subclass outputs
• CCM axis handles both setpoint-derived and linspace-derived energy lists

---

Phase 4: Cleanup + Full Migration (#89)

Dependencies: Phases 1-3 complete (all analysis paths ported)

Scan analysis

• reduce_det_scanvar: bin detector by arbitrary scan variable
• Replaces: ScanAnalysis_1D, ScanAnalysis_1D_XES

PostProcessing integration

• Port XSpect_PostProcessing.py (exponential fitting, kinetics extraction)
• Determine: registered reduction steps vs. standalone post-hoc utilities

Visualization refactor

• XSpect_Visualization.py and XSpect_Diagnostics.py read from run.results dict
• No more getattr(run, key) pattern

Deletion

• Remove XSpect_Controller.py entirely (all 6 subclasses dead)
• Remove backwards-compat shims from init.py

Acceptance criteria:

• All current analysis workflows expressible as YAML
• Visualization works with new results format
• No remaining imports from old controller subclasses
• Old controller file deleted

---

Dependency Graph

1A --> 1B --> 1C --> Phase 2 --> Phase 3 --> Phase 4
                       |                       |
                       +-- shares time_binning -+

Critical path: registry.py (1A) -> step registration (1B) -> integration test (1C). Everything after Phase 1 is incremental step additions following the same pattern.

Inter-phase dependencies:

• Phase 2 depends on Phase 1C (pipeline must be proven end-to-end before adding complexity)
• Phase 3 depends on Phase 2 (time_binning step is shared; reduction patterns established in Phase 2 reused in Phase 3)
• Phase 4 depends on Phases 1-3 (cannot delete old code until all paths are ported)

Within Phase 1:

• 1B depends on 1A (registry must exist)
• 1C depends on 1A + 1B (needs both infrastructure and registered steps)
• 1A model/ refactoring is independent of analysis/registry.py, but both must land before 1B starts

---

Design Decisions

Decision	Resolution
Pipeline model	Flat ordered list, top-to-bottom execution, no conditionals
Step dispatch	Registry via @register_step(name) decorator
Run state	run.results flat dict with dot-separated keys
on: field semantics	String passed to step; step interprets (mask name, detector key, or derived key)
Parallelism	Transparent to user. batch_size/cores at runtime, not in YAML
Reduction lifecycle	Pipeline.run() has two internal phases: batch-parallel pipeline, then serial reduction across all runs
Backwards compat	Old imports via shim init.py through Phases 1-3; removed in Phase 4
Test data	Small synthetic HDF5 fixture in tests/fixtures/ for unit tests
YAML validation	Steps self-describe expected args; config_parser raises on unknown step names or missing required fields
Notebook interface	Pipeline.from_yaml(path).run(cores=16, batch_size=2000)
Compute resources	Out of YAML, passed at runtime (notebooks on HPC nodes)
Resumability	Future feature, not in scope for Phases 1-4

---

What disappears

The following classes are replaced by different YAML files + one generic Pipeline class:

• XESBatchAnalysis
• XESBatchAnalysisRotation
• XASBatchAnalysis
• XASBatchAnalysis_1D_ccm
• XASBatchAnalysis_1D_time
• ScanAnalysis_1D
• ScanAnalysis_1D_XES

---

Sub-issues

• Phase 1A: Core infrastructure — registry, model, controller skeleton #84 — Phase 1A: Core infrastructure (registry, model, controller skeleton)
• Phase 1B: Register static XES analysis steps #85 — Phase 1B: Register static XES analysis steps
• Phase 1C: Static XES end-to-end YAML execution #86 — Phase 1C: Static XES end-to-end YAML execution
• Phase 2: Time-resolved XES pipeline steps #87 — Phase 2: Time-resolved XES pipeline steps
• Phase 3: XAS pipeline steps #88 — Phase 3: XAS pipeline steps
• Phase 4: Scan analysis, PostProcessing, and Visualization integration #89 — Phase 4: Scan analysis, PostProcessing, Visualization integration

Execution order

#84 -> #85 -> #86 (MVP complete) -> #87 -> #88 -> #89

[lg345]

Sub-issues

• Phase 1A: Core infrastructure — registry, model, controller skeleton #84 — Phase 1: Core infrastructure (registry, model, controller skeleton)
• Phase 1B: Register static XES analysis steps #85 — Phase 1: Register static XES analysis steps
• Phase 1C: Static XES end-to-end YAML execution #86 — Phase 1: Static XES end-to-end YAML execution
• Phase 2: Time-resolved XES pipeline steps #87 — Phase 2: Time-resolved XES pipeline steps
• Phase 3: XAS pipeline steps #88 — Phase 3: XAS pipeline steps
• Phase 4: Scan analysis, PostProcessing, and Visualization integration #89 — Phase 4: Scan analysis, PostProcessing, Visualization integration

Execution order

#84 → #85 → #86 (MVP complete) → #87 → #88 → #89