fix the sidecar image build, which could not build GPU support - #15
Merged
Conversation
The image did not build at all. Two independent reasons:
- The build stage was golang:1.21 while go.mod requires go >= 1.24.0.
- It built with CGO_ENABLED=0, but pkg/docker/gpustrategies uses
github.com/NVIDIA/go-nvml, which is a cgo package.
Fixing only the second one is not enough: go-nvml uses RTLD_DEEPBIND, a
glibc-only extension, so it cannot be compiled against musl either
("could not determine what C.RTLD_DEEPBIND refers to"). That rules out both
the Alpine golang image and the Alpine based docker:*-dind runtime image this
used to ship, and there is no official Debian based docker:dind image.
So both stages are now Debian based: golang:1.24 to build with CGO_ENABLED=1,
and debian:bookworm-slim with Docker installed from Docker's apt repository to
run dockerd alongside the sidecar.
Two things that fall out of the new base image:
- VOLUME /var/lib/docker is now declared explicitly. dockerd cannot stack
overlayfs on the container's own overlay filesystem; docker:dind declared
this for us, and without it every container started inside the DIND fails to
mount its rootfs.
- The startup script is written with printf instead of echo -e: /bin/sh is
dash here, which would write the "-e" out literally.
The separate bash stage is gone, since bash is part of the base image now.
Verified by building the image and running it: dockerd starts, a container
runs inside the DIND, and the sidecar binary starts and gets past GPU
discovery without a driver present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The image did not build at all. Two independent reasons:
Fixing only the second one is not enough: go-nvml uses RTLD_DEEPBIND, a glibc-only extension, so it cannot be compiled against musl either ("could not determine what C.RTLD_DEEPBIND refers to"). That rules out both the Alpine golang image and the Alpine based docker:*-dind runtime image this used to ship, and there is no official Debian based docker:dind image.
So both stages are now Debian based: golang:1.24 to build with CGO_ENABLED=1, and debian:bookworm-slim with Docker installed from Docker's apt repository to run dockerd alongside the sidecar.
Two things that fall out of the new base image:
The separate bash stage is gone, since bash is part of the base image now.
Verified by building the image and running it: dockerd starts, a container runs inside the DIND, and the sidecar binary starts and gets past GPU discovery without a driver present.