Pull stable v1.0.0 to main branch. - #15
Merged
Merged
Conversation
Update dev branch to latest
Added os.makedirs with exist_ok=True to create the mask directory if it does not exist, preventing errors when saving the mask file. Also updated file path handling to use os.path.join for better cross-platform compatibility.
…. Backwards compatible for old notebooks.
… return a scalar or str if the return variable is of len(1).
…numbers type check
Pull from v1.0.0 to dev (v1.0.0-beta.1)
Update pyproject.toml dependencies: drop the upper bound on numpy (replace "numpy<2" with "numpy") to allow numpy 2.x, and remove "pathlib" from dependencies since it is included in the Python standard library (project requires Python >=3.9). Other dependencies unchanged.
Add an HDF5 data file (attenuation_lengths.h5) and a helper _load_attenuation_table() that reads E_values and length arrays via h5py. Replace hard-coded attenuation tables in Si, Al, Be and KaptonHN attenuation-length functions with HDF5-backed loading and keep existing spline interpolation/units conversion. Also add input validation in correction_factor (cast q_arr to numpy array and require keV > 0) and introduce h5py as a dependency.
Add input validation and error reporting for _iam_loader and iam_elastic_pattern (TypeError for invalid system types; ensure q_arr is a 1D float array). Replace the O(N^2) double-loop Debye sum with a vectorized implementation using np.triu_indices and broadcasting to compute molecular interference (significant performance improvement). Vectorize per-element Compton-shell profile evaluation in iam_compton_spectrum by interpolating over the flattened pz grid and applying a liftoff mask for all theta/E at once. Minor comment clarifications. These changes improve robustness and speed for larger systems and q grids.
Add robust input checks and conversions in utils.py: compute_q_map now validates that x and y have matching shapes and that keV is positive (and converts x/y to arrays); azimuthalBinning now converts img/x/y to arrays, ensures matching shapes and positive keV. Replace immediate loading of Jungfrau4M.h5 with a _LazyJ4M descriptor to defer I/O until first access (includes __getattr__ and __repr__). These changes prevent invalid inputs and avoid expensive file loading at import time.
Only load required datasets from HDF5 files (builds needed_keys and always includes 'lightStatus/xray'), and optionally print dataset names/shapes when verbose. Replace full-file greedy loading with per-key reads to reduce memory and improve performance. Refactor concatenation logic to use chunk lists and np.concatenate, simplify run_indicator construction, and handle the special epics gasCell pressure case by fetching and interpolating EPICS data into chunks before concatenation. Misc: handle KeyError for missing keys, small style/whitespace fixes, progress bar formatting, and simplify runNumToString using str.zfill(4).
Relax numpy constraint, optimize HDF5 loading, and validate inputs, optimize iam calculations
Handle cases where I_q_elastic and/or I_q_inelastic are missing by conditionally reading and adding them to the returned SimpleNamespace (build attrs dict and return SimpleNamespace(**attrs)). Update the generated docstring to include elastic/inelastic sections only when those arrays are present. Adjust the type stub to import Optional and mark I_q_elastic and I_q_inelastic as Optional to reflect their possible absence.
Update masking.py to include 'Sums/jungfrau4M_calib' in the _KEYS_TO_SUM list so this calibration key is included in summation operations alongside the existing threshold and dropped keys. This ensures the calibration sum is aggregated consistently during masking/calibration processing. Mainly this is included in case the dark run was taken before the x-rays were delivered. In this case the evt codes are wrong and they will need to be flipped manually in the notebook. In addition, the calib_dropped key will not contain any data since the evt codes were wrong. In this case, this key would need to be used. Override this manually if this is the case.
Replace the archImport boolean with an archPVs parameter (str or list) and extend combineRuns to fetch arbitrary EPICS PVs from the archive and interpolate them onto run unixTime. The docstring was updated with usage and a note showing how to save unixTime in producer scripts. Removed the previous special-case handling for epicsUser/gasCell_pressure and simplified key concatenation logic; each requested PV is now loaded per-run, interpolated using previous-value interpolation, and stored in data_combined under the PV name. Note: this changes the function API (archImport -> archPVs) and requires unixTime to be present in the input HDF5 data.
Move legacy test files from tests/ into tests/old to keep the test root clean. Renamed/moved several .xyz files (DMP_D, DMP_L1, DMP_L2, DMP_ground) and multiple test notebooks (azav_test, calib_test, geometries_test, iam_oriented_test, iam_pattern_test, patterns_test). No content changes, only path updates.
Add tests/tests_utils.py containing a comprehensive pytest suite for xrayscatteringtools.utils. Covers unit conversions (au<->invÅ, keV<->Å, q<->theta) including round-trip checks and array handling; element symbol/number lookups and dictionary consistency; molecule translation and rotation (identity, 90° rotations, distance preservation); compute_q_map behavior, shapes, symmetry and error handling; azimuthalBinning functionality (output shapes, phi bins, masking, dark/gain corrections, rBin, custom edges, thresholding, and validation of input errors); compress_ranges formatting and edge cases; and basic checks for the J4M lazy loader. Tests assert correct results and appropriate exceptions for invalid inputs.
Add comprehensive pytest suite for xrayscatteringtools.calib.timetool_calibration (tests/test_timetool_calibration.py). Covers fast_erf_fit (behavior, edge cases, normalization, thresholds, types), add_calibration_to_yaml (YAML creation/appending, inf parsing, key handling, file errors), and apply_timetool_correction (single and multi-run corrections, broadcasting, shape/argument validation, return types). Includes fixtures for synthetic erf data and temporary YAML files.
Add comprehensive tests for xrayscatteringtools.calib.scattering_corrections in tests/test_scattering_corrections.py. The tests validate attenuation-length functions, per-material correction functions (Si, KaptonHN, Al, Be), cell and total correction_factor behavior (including input validation and custom thicknesses), and J4M_efficiency. Also update pyproject.toml to filter the expected RuntimeWarning from xrayscatteringtools.calib.scattering_corrections during pytest runs.
Introduce tests/tests_plotting.py covering xrayscatteringtools.plotting. Adds unit tests for compute_pixel_edges (shape, uniform grids, corner extrapolation, monotonicity, minimum size), edges_from_centers (edge lengths, uniform/non-uniform spacing, error cases, bracketing), and plotting helpers plot_jungfrau and plot_j4m (QuadMesh return, axis handling, aspect, vmin/vmax behavior, and J4M coordinate fallback using a mock). Tests use the non-interactive Agg backend and ensure figures are closed.
Introduce tests/test_patterns.py covering xrayscatteringtools.theory.patterns. Tests validate lazy-loading of all exported pattern objects, required attributes (q, I_q, molecule, method, basis_set, n_electrons), q/I_q shapes and properties (non-negative, sorted), optional elastic/inelastic arrays, and presence/length of docstrings. Adds __dir__ checks, unit tests for _make_default_obj using HDF5 fixtures (with and without elastic/inelastic datasets), and tests for _make_default_docstring including unit conversion text. Also includes consistency checks across shipped patterns: elastic+inelastic equals total, positive n_electrons, and non-empty molecule names. Uses pytest, numpy and h5py.
Introduce a comprehensive test suite (tests/test_masking.py) for xrayscatteringtools.calib.masking. The tests use lightweight fake J4M geometry and mocked combineRuns/plotting to avoid heavy LCLS/psana dependencies, and cover module constants, MaskMaker initialization/validation, dark/background processing, polygon masking, sample averaging, mask combination, process_sample (smoke), diagnose_q_bins (smoke), and a backward-compatible alias. Fixtures use a small synthetic detector layout for fast execution and exercise both behavior and edge cases (bounds checks, masking propagation, shapes and dtypes).
Introduce comprehensive pytest suite for the io module. Tests cover runNumToString formatting, XYZ I/O (read/write round-trips, atomic number handling, sample file check), HDF5 helpers (is_leaf, get_leaves, get_tree with verbose and saveto behaviors), YAML config helpers (get_config, get_config_for_runs, get_data_paths including boundary and error cases), and MOL file parsing (metadata, atoms, coordinates, bonds, and properties). Uses temporary files/fixtures, h5py and yaml; skips the bundled SF6 test file if it's not present.
Introduce tests/test_iam.py with comprehensive unit tests for the IAM module. Covers _iam_loader (XYZ, SimpleNamespace and MOL input), iam_elastic_pattern, iam_inelastic_pattern, iam_total_pattern, their oriented variants, and iam_compton_spectrum. Includes fixtures (SF6 geometry, q and phi arrays), and asserts on output shapes, positivity/non-negativity, consistency between XYZ and namespace inputs, physical behavior (elastic decrease at large q, forward peak, inelastic increase with q), oriented-pattern properties (phi constancy for inelastic, total = elastic + inelastic), and Compton spectrum dimensionality and non-negativity. Tests skip if the optional sf6_test.xyz is missing.
Add comprehensive unit tests for xrayscatteringtools.calib.geometry_calibration. Tests cover thompson_correction, geometry_correction, geometry_correction_units, the model function, and run_geometry_calibration. Assertions verify shapes, value bounds, symmetry, known analytic values, behavior under rotations and parameter changes, pixel-area scaling, and that run_geometry_calibration can recover synthetic geometry parameters from a generated dataset. Includes small helpers (flat theory spline and _make_detector) and uses InterpolatedUnivariateSpline for synthetic theory patterns.
Introduce comprehensive tests for xrayscatteringtools.theory.geometries. Tests cover module-level lazy loading of each entry in __all__ (attribute presence, geometry shape, atom counts, electrons, molecule string, and docstrings), __dir__ behavior, and helpers _make_default_obj (HDF5 reading into a SimpleNamespace) and _make_default_docstring (ensures docstring contains expected sections and metadata). Uses pytest, numpy and h5py to validate data and edge cases.
Create a CI workflow (.github/workflows/tests.yml) that runs the test suite on push and pull requests to main. It uses a Python matrix (3.9–3.12), checks out the repo, sets up Python (actions/setup-python@v5), installs dev dependencies via pip install -e ".[dev]", and runs pytest against the tests/ directory.
Add CI badge and expand the README with a clearer project scope and a more detailed feature list (I/O, plotting, calibration, theory, utilities). Update installation instructions (PyPI, source install, dev extras) and state Python ≥ 3.9. Flesh out the Modules Overview with concrete functions and submodules for io, plotting, utils, calib, and theory. Add testing instructions and note GitHub Actions coverage. Minor edits: update LICENSE link and fix acknowledgments/contact link.
Update pyproject.toml to set the project version from 1.0.0-beta.2 to 1.0.0, marking the stable release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.0.0
This release marks the first stable version of
xrayscatteringtools, adding a comprehensive test suite, CI/CD infrastructure, and significant improvements to the calibration, masking, and I/O modules.Highlights
fast_erf_fit,apply_timetool_correction, YAML persistenceBreaking Changes
mask_makerclass renamed toMaskMaker(old name kept as alias for backward compatibility)numpy<2constraint removed — now compatible with NumPy 2.xpathlibremoved from dependencies (stdlib since Python 3.4)Calibration & Corrections
MaskMakerwith typed parameters, input validation,__repr__, and apoly_maskattribute for polygon-based region exclusionapply_polygon_mask()for interactive or scripted polygon masking viamatplotlib.path.Pathdiagnose_q_bins()for previewing per-ring statistics before committing to the interactiveprocess_sampleloopprocess_samplenow applies a Thompson polarization correction before per-ring analysiscompute_q_map()utility for computing momentum-transfer maps from detector geometryattenuation_lengths.h5filetimetool_calibrationmodule:fast_erf_fitfor edge detection,apply_timetool_correctionfor per-shot arrival-time correction,add_calibration_to_yamlfor persisting calibration parametersedges_from_centersfromplottingandcompute_q_mapfromutilsin the top-level__init__.py'Sums/jungfrau4M_calib'to_KEYS_TO_SUMin masking constantsI/O & Data
combineRunsHDF5 loading and added EPICS archive PV import supportJ4Mgeometrytheory.iamI_q_elasticandI_q_inelasticdatasets optional intheory.patternsPackaging & Dependencies
0.3.0to1.0.0numpyconstraint (removed<2pin)pathlibdependencyruamel.yamlto dependenciesfilterwarningsinpyproject.tomlfor expectedRuntimeWarningfromscattering_correctionsTesting & CI
.github/workflows/tests.yml) — runs on push/PR tomain, Python 3.9–3.12 matrixtest_utils(61),test_plotting(23),test_io(37),test_iam(29),test_patterns(38),test_geometries(15),test_geometry_calibration(21),test_scattering_corrections(45),test_timetool_calibration(45),test_masking(46)tests/old/Documentation & Metadata
README.md: CI badge, accurate module reference, proper install instructions, testing sectionLICENSEcopyright to 2025–2026