Problem
When process_batch in src/sampleworks/eval/generate_synthetic_sf.py and src/sampleworks/eval/generate_synthetic_density.py is called with a CUDA device and n_jobs > 1, each worker process spawns a separate CUDA context. Depending on available GPU memory, this can cause OOM errors or severe performance degradation as multiple workers compete for the same GPU(s).
Suggested approach
- Allow the caller to specify a more granular device (e.g.
cuda:0, cuda:1) per worker, or
- Detect GPU usage and warn/clamp
n_jobs accordingly if CUDA device is used
Context
Flagged during code review of PR #234 (feat(synthetic-sf): add script to generate synthetic structure factors).
Also applicable to the density generation script.
Backlinks:
Requested by @marcuscollins — help from engineers appreciated.
Problem
When
process_batchinsrc/sampleworks/eval/generate_synthetic_sf.pyandsrc/sampleworks/eval/generate_synthetic_density.pyis called with a CUDAdeviceandn_jobs > 1, each worker process spawns a separate CUDA context. Depending on available GPU memory, this can cause OOM errors or severe performance degradation as multiple workers compete for the same GPU(s).Suggested approach
cuda:0,cuda:1) per worker, orn_jobsaccordingly if CUDA device is usedContext
Flagged during code review of PR #234 (feat(synthetic-sf): add script to generate synthetic structure factors).
Also applicable to the density generation script.
Backlinks:
Requested by @marcuscollins — help from engineers appreciated.