Reserve memory for DD calibration h5merge - #147
Conversation
Reserve memory for DD calibration h5merge
|
@jurjen93 how many directions did you end up merging for EN1? To get a feel for how much leeway to give the memory. |
For EN1 I had 24 directions, for EDFN I have 41 directions |
|
For EDFN I merged the Dutch DD calibration solutions with the Dutch+Int DD calibration solutions. Because of the higher time/freq resolution from the Dutch DD calibration solutions, the h5parm becomes for 41 directions around 103GB. So, maybe constrain it to 150 GB, to be sure? (I dont think we will have more than 60 directions for a field) |
| - class: DockerRequirement | ||
| dockerPull: vlbi-cwl | ||
| - class: ResourceRequirement | ||
| ramMin: 45000 |
There was a problem hiding this comment.
Suggestion: put this on 150000. My h5parm is for example 100 GB with ~40 directions, so for example ~60 directions you would need ~150 GB.
2 participants
This PR adds a memory allocation to the merging of all DD solutions. For my run Toil reports 21.61 GB used, but Slurm reports a max RSS of 44017964K, so I went with the Slurm value to be on the safe side as I don't know what Toil's polling frequency is. This is with 37 directions.