Conversation
…dex for the darks (eg below the first 10 time bins use that as the dark spectrum)
…ization also IRF fitting that is dubious right now but fits something
merging master to the current experiment to capture the recent PR for the dark shot changes.
…rence_spectrum-should-have-a-feature-to-make-difference-based-on-indices 21 xspectvisualizationplot 2d difference spectrum should have a feature to make difference based on indices
…. Scaled fast_delay_key values by 1e12 in time_binning method in XSpect_Analysis.py. Created a current clean and working notebook XSpect_XES_dehe_on_demand_05022025.ipynb
…n_demand, etc. Renamed the notebooks so that they are unique and they don't conflict with merges from others. It would be good to come up with a system/protocol for git and notebook.
…heritance chain. It doesn't seem to break anything. I just think it is neater that way.
Merging code changes from Dehe for Follmer experiment
Mfx10089224
…eys are unique and not duplicates in the final set of attributed.
This commit introduces a new method, `primary_analysis_static_laser`, specifically designed for analyzing data with static laser conditions. This method now separates shots based on laser presence and applies hitfinding and spatial reduction steps individually to the resulting shot subsets: 'epix', 'epix_simultaneous_laser', and 'epix_xray_not_laser'. This allows for a more granular analysis of laser-related effects. Key changes: * Added `primary_analysis_static_laser` method for laser analysis. * Modified the laser analysis method to filter shots and handle hitfinding, transposition, patching, rotation and spatial reduction for different shot types to enable laser-specific filtering and analysis. * Updated `keys_to_save` to include results from laser-specific analysis.
…tra (i.e. no laser)
…his at the object level
…zation taking the absolute value of the trapz because the energy axis direction changes the values of the trapezoidal integrals.
rotation_diagnostic — finds optimal detector tilt angle by running the pipeline on a subset of shots and inspecting the 2D sum image with overlaid rotation angle estimates from XSpectDetectorProcessor. Fixed angles (-2.0° / -1.6°) were determined here and hardcoded into YAMLs. pixel_patch_diagnostic — compares raw vs patched column profiles for epix100_0 and epix100_1, visualises the ASIC gap positions and the effect of auto_detect patching on the baseline ratio and z-score. seer_shot_browser — per-shot SEER image browser: scan forward/backward through individual shots to inspect photon distributions and confirm the ASIC gap correction is working correctly. seer_vs_xrt — quantifies SEER as an alternative incident-energy monitor for stochastic RIXS. Computes correlation between SEER and XRT per- shot total-intensity and center-of-mass energy. Conclusion: SEER tracks intensity well (r=0.90) but has low COM energy correlation (0.41) because it encodes sample transmission/emission, not purely the incident beam. Gap patching dramatically improves the spook AtA condition number. spook_rixs — stochastic RIXS extraction using the spook library: builds A (XRT hproj) and B (epix_spec per-shot spectra) matrices and solves for the RIXS cross-section X via regularised least squares. droplet_spook_rixs — same as spook_rixs but using photon-counted spectra from droplet_reconstruction instead of raw ADU integration. droplet_spook_transmission — stochastic RIXS using SEER (transmission geometry) as the B matrix; photon-counted for improved S/N. rixs_plane — static RIXS plane visualisation for the DCCM scan run.
regenerate_run0079.py — runs mfx101609126_pershot_xes.yaml (standard ADU integration) and saves per-shot spectra to examples/results/; used as the baseline to compare against droplet-reconstructed spectra. regenerate_droplet_pershot.py — runs mfx101609126_droplet_pershot_xes.yaml for runs 78-82 sequentially, saving one HDF5 per run to results/. Measures reconstruction throughput (shots/sec) and reports memory usage. droplet_reconstruct.py — standalone utility script that reconstructs a single run's sparse photon data without the full pipeline framework; used for rapid iteration and debugging of the reconstruction step. droplet_reconstruction.ipynb — step-by-step walkthrough of the sparse format: reads raw HDF5, shows fixed-length vs variable-length layouts, runs _scatter and _scatter_roi helpers manually, compares output images against the pre-processed ROI_area dataset for bit-identity verification. droplet_recon_pipeline_eval.ipynb — end-to-end comparison of ADU integration vs photon-counting via the pipeline API. Runs both YAMLs on run 79, overlays 1D emission spectra, and evaluates peak S/N ratios to quantify the photon-counting improvement for spook inputs.
…c assets mfx101080524_static_xes.yaml: remove hitfinding step. The hit-finder was removing shots that had low per-frame median signal, which is expected for single-photon-level XES data. Removing it recovers the correct shot count and avoids discarding weak-signal runs. seer_full_detector_run78.png: full-panel SEER detector image from run 78 showing the spatial distribution of signal across all ASIC tiles; used in rotation_diagnostic.ipynb and seer_shot_browser.ipynb to orient the cross-dispersion ROI selection. c.ipynb: scratch notebook for static RIXS pipeline diagnostics (run 97); shows RIXS plane shape, energy range, and 2D sum-image ROI inspection. xcs101591326.ipynb, Untitled.ipynb: updated experiment analysis notebooks.
…MFX pipelines Droplet reconstruction, patch_pixels auto-detect, XAS fixes, and XCS/MFX pipelines
…, and analysis notebooks
Adds the full mfx102101026 (ferricyanide) static XES analysis workflow and the
supporting XSpect engine changes needed to run it at scale, plus the earlier
mfx100895324 static-XES artifacts.
XSpect engine changes
----------------------
* model/run.py, controller/{config_parser,pipeline,batch_manager}.py:
Add an import-time row-range crop for detector keys (`row_range: [start, end]`
in the YAML `detector_keys` block). The slice is applied at HDF5 read time in
both the single-run (`load_run_key_delayed`) and batched (`_load_batch_data`)
paths, so the full detector frame is never materialized. For epix100_0 this
cuts per-batch memory ~5-7x and was necessary to stop the pipeline dying in
silent OOM. The crop origin is recorded on the run (`run._row_offset`) so that
downstream ROIs stay in absolute (full-frame) coordinates.
* analysis/spectroscopy.py:
- reduce_detector_spatial now auto-translates absolute ROI coordinates into
the cropped-array frame using the recorded row_range offset (stripping
_reduced/_ROI_* suffixes so derived keys inherit the parent offset). This
fixes ROIs silently reading out of bounds (Ka summing to zero) after a crop.
- hitfinding gains an absolute `min_sum` mode to reject genuinely dark shots
(sum == 0 after ADU threshold); the old relative median-cutoff mode is kept
as a fallback for signal-dominated data.
* controller/pipeline.py, pipeline_runner.py, __init__.py:
Add a real `logging`-based progress system on the "XSpect" logger, exposed via
`enable_logging(level, log_file=None)` (stderr and/or file). The batched path
now reports per-batch completion via imap_unordered and raises an explicit
OOM-hint error if a worker dies, replacing the previous silent hang. Per-step
debug logging and full tracebacks are emitted from run_pipeline.
mfx102101026 experiment (experiments/mfx102101026/)
---------------------------------------------------
* mfx102101026_static_xes.yaml: static Fe Ka/Kb pipeline on epix100_0
(transpose=true, row_range crop, hitfinding min_sum, ferricyanide ROIs).
* mfx102101026_static_xes_visualization.ipynb: per-run pipeline + diagnostics.
* mfx102101026_aggregate_xes.ipynb: cross-run aggregation from saved results.
* mfx102101026_ferricyanide_reference.ipynb: aggregates runs 9+10, minimum
subtraction then area normalization, saves the ferric (Fe3+) reference.
* mfx102101026_run9_baseline_speciation.ipynb: crop -> baseline -> normalize ->
speciation on the fixed windows (Ka 6380-6410, Kb 7031-7075 eV); baseline
subtraction currently disabled (passthrough) pending a robust routine.
* mfx102101026_speciation_simple.ipynb: non-expert runner. User lists runs;
runs are XSpect-processed only if not already cached (per-run HDF5), then
combined -> min-subtract -> normalize -> speciation vs the ferric reference,
with a difference spectrum and an eLog beamline-summary section (compiles the
figures, labeled by run, into stats/summary/XSpect_speciation via SummaryPost).
* mfx102101026_droplet_visualization.ipynb: droplet sparse-data viewer. Reads
only ROI droplets by streaming contiguous slabs of the first N sparse columns
(avoids the 28 GB full-array read that hung the notebook); documents that
droplet coordinates are in the raw, non-transposed detector frame.
* runs.csv: run bookkeeping; runs 4/5/8 flagged as no-beam, run 9 first 30 Hz.
* results/*.h5: cached per-run and aggregate spectra, ferric reference.
* FeIIICN6_*.dat: exported Ka/Kb reference spectra.
mfx100895324 (experiments/mfx100895324/) and examples/
------------------------------------------------------
Adds the static-XES notebooks, YAML, runs.csv, manifest/README, and saved
results that the mfx102101026 work was derived from, plus a spike-finder
example notebook.
Note: the LUTE smalldata configs and epix common-mode diagnosis for
mfx102101026 live in the experiment data area (not this repo); summary_post.py
lives alongside the notebooks in results/lbgee/ for the same reason.
…r mfx102101026 Adds a repository skill documenting how to stand up a new XES experiment end-to-end, plus the droplet-reconstruction pieces it references. skills/XSpect-setup-XES/SKILL.md - Phased guide (experiment-type triage -> LUTE smalldata pipeline -> geometry discovery -> XSpect YAML -> diagnostic/analysis notebooks -> issue tracking). - Decision questions to classify the analysis: static, time-resolved pump-probe, CCM-scanned/RIXS, or droplet/photon-counted XES. - YAML templates for each type, memory (row_range) guidance, the ePix100 common-mode gotcha, droplet vs droplet2photon distinction, and the LUTE droplet2photon template gap (issue #97). - Guardrails: never invent step names, verify units (ADU vs keV) and the transpose/ROI coordinate frame, confirm beam-on before debugging. - README links to the skill and the YAML pipeline guide. experiments/mfx102101026/ - mfx102101026_droplet_recon_xes.yaml: native photon-counted workflow using the droplet_reconstruction step (ROI applied on sparse arrays) -> union -> sum -> rotate -> reduce, modeled on the working mfx101609126 droplet pipeline. - droplet_visualization.ipynb: Section 7 reconstruction now uses the native XSpect droplet_reconstruction step instead of an external helper script.
…ing conventions Adds Phase 0.5 (Identify the HDF5 keys) and strengthens the guardrails so the skill enumerates datasets and asks the user when a role is ambiguous rather than guessing paths. - Role -> naming conventions: emission = ePix100 (epix100_0/1, epix_0/1 and variants); scattering = epix10k2m / jungfrau16m (usually not the XES signal); I0 monitor = ipm<N>/sum in standard hutches but MfxDg*BmMon/... at MFX; lightStatus masks, timing keys, CCM scan variable, von Hamos geometry PVs. - Ambiguity protocol: 0 candidates -> ask / offer to skip; >1 -> list with shapes and ask; always echo the final key->role mapping for confirmation. - Notes that with two ePix100s (spectrometer + SEER) the agent must ask which is which instead of assuming _0 is the spectrometer.
…ring Zeros low-variance detector pixels via sklearn VarianceThreshold instead of a hand-tuned ADU cutoff. Pixels that barely change across shots (dead, static hot, constant background) are dropped; signal-bearing pixels are kept. Writes back to the same detector key and stores the retained mask as <key>_variance_mask. Handles 3D and 2D detectors. Closes #36
…itecture Brings in the mfx102101026 static + droplet Fe Kα/Kβ XES analysis and the XSpect engine features developed for it: - Import-time detector row_range crop with automatic ROI offset translation (model/run.py, controller/batch_manager.py, analysis/spectroscopy.py) — the memory fix that prevents OOM in batched runs. - hitfinding min_sum absolute mode to reject dark shots. - logging system with enable_logging() and per-batch progress (controller). - experiments/mfx102101026/: static XES YAML, droplet recon YAML, and visualization/aggregate/reference/speciation/droplet notebooks. - experiments/mfx100895324/: predecessor static-XES artifacts. - skills/XSpect-setup-XES/: repository skill for standing up a new XES experiment (LUTE + XSpect YAML + diagnostics), with HDF5 key identification. Run 26 tracking: experiment #98, tracked under #99; open dev items #97, #96, #70, #55, #45.
…ne-architecture # Conflicts: # XSpect_XES_mfx101080524.ipynb
YAML-driven pipeline ar Here's an extended merge description you can paste into the GitHub merge box. Merge YAML-driven pipeline architecture into master (#100) Replaces the monolithic controller with a declarative pipeline. Analysis is defined in a YAML config as an ordered list of steps; a registry maps each step name to a stateless function step(run, **kwargs). Model, controller, and analysis layers are separated so the same steps run across XES, XAS, RIXS, and droplet workflows. Architecture (issue #82, Phase 1A) - Step and reduction registry (XSpect/analysis/registry.py) with decorator registration and lookup by name. - Config parser that reads the data and pipeline sections of a YAML file. - Batch manager with multiprocessing, scalar pre-pipeline pass, max_shots limit, and absolute HDF5 index handling. - Pipeline runner that executes steps in order and writes results onto the run. Registered steps (Phases 1B/1C and time-resolved XES: #85, #86, #87) - Loading: load_run_keys, load_detector (ROI, transpose, combine). - Shot and pixel filtering: filter_shots, filter_detector_adu, filter_detector_variance. - Combination: union_shots, separate_shots. - Spatial and temporal reduction, time binning, CCM binning (3D detector support, NaN handling). - Energy calibration: make_energy_axis, make_ccm_axis. - Normalization, rotation detection (find_rotation_angle), rotate_detector. New capabilities - droplet_reconstruction for sparse photon-counting HDF5 layouts. - patch_pixels with auto-detection (ratio and z-score methods) and vectorized polynomial fitting. - filter_detector_variance: sklearn VarianceThreshold as a data-driven alternative to a hand-tuned ADU cutoff. Zeros pixels whose value barely changes across shots and stores the retained mask as <key>_variance_mask. Closes #36. Examples and docs - YAML configs and analysis notebooks for mfx101080524, mfx101609126, mfxl1027922, xcs101591326, and mfx102101026 beamtimes. - User guide at docs/YAML_PIPELINE_GUIDE.md. - New XSpect-setup-XES skill for HDF5 key identification and detector naming. Testing - Regression suite with saved reference arrays for static and ultrafast XES. - Unit tests for pipeline steps (32 pass locally for the step and spectroscopy suites). - Pre-existing failures in test_xas_steps.py (CCM off-by-one) and test_batch_manager.py are unrelated to this branch and predate the merge. Compatibility and conflict resolution - Old imports still work (from XSpect.XSpect_Analysis import spectroscopy_run). - origin/master (XAS foil-scan and derivative-analyzer work, PR #90) was merged into the branch first. One conflict resolved: the root-level XSpect_XES_mfx101080524.ipynb was deleted per master's reorg; the canonical copy remains at examples/XSpect_XES_mfx101080524.ipynb. XSpect_Analysis.py auto-merged and should get a review pass.chitecture with MVC separation
Quarterly review (2026-04-16 to 2026-07-16) covering the YAML pipeline overhaul, new analysis steps, bug fixes, and LUTE work for discussion. README now describes the model/controller/analysis layers, the step registry pattern, data flow, and per-mode use cases with example configs.
Move the four pipeline-YAML templates out of SKILL.md into templates/, reached by a Phase 3 pointer keyed to the experiment type, so a run reads only the one it needs. Thin the Guardrails section to a failure-mode checklist that points at each rule's canonical phase instead of restating it. Trim the description to identity plus triggers and add the droplet type.
Port the MkDocs Material + mkdocstrings site off the stranded jtb branch onto master and retire the parallel Sphinx/Read the Docs setup, so there is one documentation source. - mkdocs.yml: mkdocstrings pointed at XSpect.model/controller/analysis via new source pages, plus the legacy top-level modules; nav includes the YAML pipeline guide. Drop the logo/favicon refs (files never existed in source). - .github/workflows/docs.yml: build and gh-deploy on push to master (the old workflow only triggered on jtb, so the live site froze at 2025-10-10). Install requirements.txt so mkdocstrings can import the package in CI. - Remove docs/_build, conf.py, *.rst, Makefile/make.bat, docs/requirements.txt, and .readthedocs.yaml. - Fix index.md links (.html -> .md) and a pipeline.py docstring that griffe read as a bogus parameter. Build verified locally. Refs #101
common_mode_correction subtracts per-row, per-column, or per-bank electronic baseline estimated from a signal-free reference band, on 3D detector data before shot reduction, preserving shape. subtract_polynomial_background fits a low-order polynomial to signal-free regions along the spatial axis and subtracts it, writing a non-destructive <key>_bkgsub. peak_mask accepts multiple ranges so several emission lines dispersed on one detector can be masked at once. Per-row weighting excludes NaNs from the fit. Reuses the vectorized weighted-polyfit approach from patch_pixels. Adds unit tests for both steps and documents them in the YAML guide and README step lists. Closes #102
…kgsub Add common_mode_correction and subtract_polynomial_background steps
Hand-written reference for all 28 steps plus combine_runs. One index page (step contract, array-shape vocabulary, master table) and four category pages: loading & filtering, detector & spatial, binning, spectra & reductions. Each entry lists what the step reads, what it writes, and every tunable parameter with its default. Wire a Step reference nav section into mkdocs.yml after the YAML guide; keep source_analysis.md as the source-linked API.
Add detailed step reference docs
Each step and the combine_runs reduction now carries a short YAML snippet showing a typical invocation, so every entry has one.
…gs page Group the guide, a new Example configs page, the step reference, and the source-linked API under one "YAML pipeline" nav section. The example configs page embeds the six analysis-mode YAML files live via snippets, so the docs track the real files in examples/.
Add YAML example to every step reference entry
Nest pipeline docs and add example configs page
Allow the YAML experiment section to specify an explicit smalldata_dir,
overriding the default /sdf/data/lcls/ds/{hutch}/{exp}/hdf5/smalldata
location search. Enables running the pipeline on locally reprocessed
smalldata (e.g. custom pedestal/droplet params) without touching the
shared experiment directory.
Estimate a per-line fluorescence background from side-band regions flanking a dispersed spectral streak (on the cross-dispersion axis) and subtract it from the signal band. Removes the isotropic fluorescence pedestal under a Von Hamos emission line while preserving the lineshape, before spatial reduction to a 1D spectrum.
Extract one EVR/timing event-code column as a per-shot boolean mask so it can be consumed by union_shots/filter_shots. Needed to gate 30 Hz beam delivery (event code 198) inside a 120 Hz DAQ, which the smalldata xray flag does not distinguish. Used by the mfx101592326 Fe Ka droplet pipeline.
mfx102101026 Fe Kα pipeline steps: event-code mask, spatial background, smalldata override, Zr calibration
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs are out of date compared to master. Need to merge master changes into jtb to build docs.