Add GPU support for containers - #122
Conversation
|
This code has been tested by running LISFLOOD-FP on a GPU-enabled container via SEPEX. The model ran with no issues. |
There was a problem hiding this comment.
I think this needs a lot of thinking, because it is a rule modification that we are making, we are saying that we will limit containers by the resources in their registration YAML.
Before this we were not setting CPU and Mem limitation at Container launch. These resources were just used for job queue to approximate how many jobs to run. This is something I decided based on how Docker CPU and Mem limitation work, which is not straight forward. So for example if I set --cpus=2, it only means that if the container tried to use more than these CPUs, Docker will kill the container. We do not want this behavior. It does not mean Docker will wait to reserve these resources. There is no guaranteed reservation with Docker.
Also we need to integrate GPU resources in configuration.
I think at most what we can safely do is pass GPU flag to Docker if a process requires GPU.
I recommend that we park this issue as feature request and discuss all the consequences there and/or for now do a feature enhancement to just pass GPU flag to docker to use GPU when it is requested by process and available on host machine.
| Resources: container.Resources(resources), | ||
| } | ||
|
|
||
| // Set device requests (for GPUs) from resources |
There was a problem hiding this comment.
Would this set just GPU or CPU as well?
| if maxMemory > 0 && p.Config.Resources.Memory > maxMemory { | ||
| return fmt.Errorf("process requires %dMB memory but max allowed is %dMB", p.Config.Resources.Memory, maxMemory) | ||
| } | ||
| if p.Config.Resources.GPUs < 0 { |
There was a problem hiding this comment.
What if requested GPUs are higher than machine requested.
|
Closing it in favor of a comprehensive planned work. |
2 participants
This PR adds a config field to allocate a GPU alongside other resources (CPU and memory). When the GPU field is greater than 1, docker processes will execute with the gpus=# flag.
(warning: this code was update by AI)