Dask improvements#38
Merged
Merged
Conversation
Now that PSC_PLOT_DASK_SCHEDULER=distributed is supported in cli.py, distributed should be a real dependency, not pip-install-it-yourself. The import in cli.py stays lazy so the threads-only default doesn't pay the tornado/zict import cost on every startup. Co-Authored-By: Claude <noreply@anthropic.com>
Extract the preprocess lambda to a module-level function so it survives pickling for dask's processes scheduler.
New PSC_PLOT_DASK_SCHEDULER env var, threaded through CONFIG so cli.py can call dask.config.set(scheduler=...) when set. Empty/unset leaves dask's default in place. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
When PSC_PLOT_DASK_SCHEDULER=distributed, spin up a LocalCluster with n_workers=num_workers, one thread per worker, real processes. Workers are persistent across the run so the per-task spawn cost amortizes. Requires 'distributed' to be pip-installed; lazy import so the import error only fires when the scheduler is explicitly requested. Co-Authored-By: Claude <noreply@anthropic.com>
prt-bin-time benches show ~2.5x wall speedup from 1 -> cpu_count threads on the particle binning workload (the case that's actually CPU-bound). No regression on field scenarios. Drops the nag-warning since most users want the parallel default. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
JamesMcClung
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add more dask config options. Also add a hamscan test and related bug fix.