Skip to content

fix the sidecar image build, which could not build GPU support - #15

Merged
Bianco95 merged 1 commit into
mainfrom
fix-dockerfile-cgo
Aug 20, 2026
Merged

fix the sidecar image build, which could not build GPU support#15
Bianco95 merged 1 commit into
mainfrom
fix-dockerfile-cgo

Conversation

@Bianco95

Copy link
Copy Markdown
Collaborator

The image did not build at all. Two independent reasons:

  • The build stage was golang:1.21 while go.mod requires go >= 1.24.0.
  • It built with CGO_ENABLED=0, but pkg/docker/gpustrategies uses github.com/NVIDIA/go-nvml, which is a cgo package.

Fixing only the second one is not enough: go-nvml uses RTLD_DEEPBIND, a glibc-only extension, so it cannot be compiled against musl either ("could not determine what C.RTLD_DEEPBIND refers to"). That rules out both the Alpine golang image and the Alpine based docker:*-dind runtime image this used to ship, and there is no official Debian based docker:dind image.

So both stages are now Debian based: golang:1.24 to build with CGO_ENABLED=1, and debian:bookworm-slim with Docker installed from Docker's apt repository to run dockerd alongside the sidecar.

Two things that fall out of the new base image:

  • VOLUME /var/lib/docker is now declared explicitly. dockerd cannot stack overlayfs on the container's own overlay filesystem; docker:dind declared this for us, and without it every container started inside the DIND fails to mount its rootfs.
  • The startup script is written with printf instead of echo -e: /bin/sh is dash here, which would write the "-e" out literally.

The separate bash stage is gone, since bash is part of the base image now.

Verified by building the image and running it: dockerd starts, a container runs inside the DIND, and the sidecar binary starts and gets past GPU discovery without a driver present.

The image did not build at all. Two independent reasons:

- The build stage was golang:1.21 while go.mod requires go >= 1.24.0.
- It built with CGO_ENABLED=0, but pkg/docker/gpustrategies uses
  github.com/NVIDIA/go-nvml, which is a cgo package.

Fixing only the second one is not enough: go-nvml uses RTLD_DEEPBIND, a
glibc-only extension, so it cannot be compiled against musl either
("could not determine what C.RTLD_DEEPBIND refers to"). That rules out both
the Alpine golang image and the Alpine based docker:*-dind runtime image this
used to ship, and there is no official Debian based docker:dind image.

So both stages are now Debian based: golang:1.24 to build with CGO_ENABLED=1,
and debian:bookworm-slim with Docker installed from Docker's apt repository to
run dockerd alongside the sidecar.

Two things that fall out of the new base image:

- VOLUME /var/lib/docker is now declared explicitly. dockerd cannot stack
  overlayfs on the container's own overlay filesystem; docker:dind declared
  this for us, and without it every container started inside the DIND fails to
  mount its rootfs.
- The startup script is written with printf instead of echo -e: /bin/sh is
  dash here, which would write the "-e" out literally.

The separate bash stage is gone, since bash is part of the base image now.

Verified by building the image and running it: dockerd starts, a container
runs inside the DIND, and the sidecar binary starts and gets past GPU
discovery without a driver present.
@Bianco95
Bianco95 merged commit 9c0345f into main Aug 20, 2026
5 of 8 checks passed
@Bianco95
Bianco95 deleted the fix-dockerfile-cgo branch August 20, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant