Add lightweight realization helpers and serialization options#18
Merged
Conversation
Add substructure_flag() to derive the model abbreviation (CDM/WDM/...) from a pyHalo realization, and strip_realization() to drop the ~59 MB realization object from a StrongLens for lightweight serialization, recording the substructure flag and leaving a truthy sentinel so has_realization semantics still hold.
Add a subhalos.serialization config key ('full' or 'lightweight') to
every config. In lightweight mode, step 03 strips the pyHalo realization
object from each subhalo'd pickle (~76% smaller) since step 04
ray-shoots from the baked-in kwargs_lens, not the realization.
make_substructure_csv reads the flag stored at strip time when the
realization is gone, else derives it via lensing.substructure_flag. The
roman_data_challenge configs opt into lightweight; the rest keep full.
The pipeline test is parametrized over both modes.
Rewrite already-written full step-03 pickles the way lightweight serialization would have, without re-running pyHalo. Resumable, idempotent (skips None / already-stripped pickles), and atomic (writes to .tmp and os.replace-s into place). Safe to run whether or not step 04 has consumed the output.
Cover substructure_flag derivation, strip_realization preserving the lensing inputs while shrinking the pickle, the sentinel round-trip and its agreement with LightweightStrongLens, image positions unchanged after stripping, and the compact_03_realizations round-trip.
Relocate the rung-1 subhalo script (byte-identical) from the shared pipeline directory to the project it belongs to.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new "lightweight" serialization mode for subhalo realizations, significantly reducing the disk space required for storing lensing simulation outputs by stripping the large pyHalo realization object after its information is baked into the lens model. The changes add support for this mode across the pipeline, including configuration, realization generation, CSV export, and a one-off script to compact existing outputs. The implementation ensures that essential metadata (the substructure flag) is preserved and that downstream steps remain compatible.
The most important changes are:
Serialization and Disk Usage Improvements:
Added a
serializationoption (fullorlightweight) to all relevant survey configuration YAML files, allowing users to choose whether to keep or strip the heavy pyHalo realization object in step 03 pickles. The "lightweight" mode reduces file size by ~76%. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]Updated
_03_generate_subhalos.pyto respect the newserializationconfig, invoking the newstrip_realizationfunction after adding the realization if "lightweight" is selected. This function drops the heavy realization and preserves the substructure flag. [1] [2] [3]Added
strip_realizationandsubstructure_flagutilities tolensing.pyfor stripping the realization and reliably extracting the substructure model abbreviation, ensuring metadata is retained.Pipeline and Data Management:
compact_03_realizations.py, a stand-alone, atomic script to retroactively strip realizations from existing step 03 pickles, making them lightweight without re-running pyHalo. This script is idempotent and safe to run at any point.Downstream Compatibility and Metadata Handling:
make_substructure_csv.pyto read the substructure flag from the new location when using lightweight serialization, ensuring CSV exports remain correct and compatible regardless of serialization mode.Other Minor Changes:
These changes together enable much more efficient storage and handling of large lensing simulation datasets, while preserving all necessary metadata for downstream analysis.