ci: build docker images from prebuilt binaries, on a distroless base - #54
Merged
Conversation
The image builds compiled the whole crate a second time inside Docker and were the two slowest jobs in the pipeline — 677s for arm64 and 530s for amd64, against 345s for the same compile in the binary job. They now copy the binary the release already builds, which drops them to a COPY plus a smoke test. That constrains glibc: symbol versioning is forward-compatible only, so a binary built on ubuntu-latest (2.39) links fine and dies at exec time on debian:bookworm-slim (2.36). The Linux binaries therefore build inside rust:1-bookworm rather than on the runner, which also lowers the glibc floor on the published tarballs. Static musl would remove the constraint entirely but isn't available here: kreuzberg embeds pdfium via include_bytes! and extracts and dlopens it at runtime, which a fully static binary cannot do. ca-certificates stays in the image for a related reason — reqwest resolves roots through rustls-native-certs, which reads the system trust store rather than carrying its own, so a scratch base would fail every handshake. Both constraints are documented in the Dockerfile. Images now build on pull requests too. Nothing else in the pipeline builds the image, so a broken Dockerfile only surfaced after merge; it costs seconds now that there is no compile in it. The darwin binaries stay main-only since no image consumes them and macOS runners bill at ten times the Linux rate. Adds linux-aarch64 to the release assets, which the arm64 image needed anyway. The cost is that the Dockerfile is no longer standalone-buildable; it carries a copy-paste one-liner for building it by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
radiosilence
force-pushed
the
ci/reuse-binaries-in-image
branch
from
July 26, 2026 19:17
95f07ed to
a08f6dd
Compare
The image build compiled the whole crate a second time inside Docker and was the slowest thing in the pipeline — 677s for arm64 and 530s for amd64, against 345s for the same compile in the binary job. With lto = true and codegen-units = 1 those are the most expensive compiles in the repo, and we paid for two of them twice per push. build-image now owns that compile outright and copies the result into the image. Keeping it in one job rather than handing off from a separate build job avoids a runner handoff and an artifact round-trip, and sidesteps upload-artifact dropping the executable bit. ubuntu-22.04 is the glibc floor, not an arbitrary pin: 2.35 against bookworm-slim's 2.36. Symbol versioning is forward-compatible only, so building on ubuntu-latest (2.39) would link fine and die at exec time in the image. It also makes the published tarballs portable further back than they were. `docker build .` still works from a clean clone. The Dockerfile declares a global ARG BIN_SOURCE=source and selects between a source-compiled stage and a copy-the-artifact stage; BuildKit only builds the stage that resolves, so CI's --build-arg BIN_SOURCE=prebuilt never touches the Rust builder. The ARG must sit before the first FROM to be usable in one, and the selection has to go through a FROM — --from takes no variable expansion. Static musl would remove the glibc question entirely but isn't available here: kreuzberg embeds pdfium via include_bytes! and extracts and dlopens it at runtime, which a fully static binary cannot do. ca-certificates stays for a related reason — reqwest resolves roots through rustls-native-certs, which reads the system trust store, so a scratch base would fail every handshake. Dropped cache-from/cache-to on the Docker build: nothing compiles in the image now, and mode=max was stuffing Rust build layers into the same repo-wide 10GB budget Swatinem/rust-cache uses, so the two evicted each other. The compile jobs now save-if main, since branch-scoped caches aren't readable from other branches and PR runs were only evicting the entry they restore from. Images build on pull requests too. Nothing else builds the image, so a broken Dockerfile only surfaced after merge. Adds linux-aarch64 to the release assets, which the arm64 image needed anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
The image build compiled the whole crate a second time inside Docker and was the slowest thing in the pipeline — 677s for arm64 and 530s for amd64, against 345s for the same compile in the binary job. With lto = true and codegen-units = 1 those are the most expensive compiles in the repo, and we paid for two of them twice per push. build-image now owns that compile outright and copies the result into the image. Keeping it in one job rather than handing off from a separate build job avoids a runner handoff and an artifact round-trip, and sidesteps upload-artifact dropping the executable bit. The runtime image is now distroless: the binary is the only thing it needs to contain, so there is no shell, no package manager and no apt layer. It is not scratch because three things rule that out, each verified against the image: ca-certificates.crt, because reqwest resolves roots through rustls-native-certs and reads the system trust store; libstdc++ and a dynamic loader, because kreuzberg embeds pdfium and dlopens it at runtime (which is also why a fully static musl build is out); and /tmp, where pdfium is extracted on first use. Runners are pinned to ubuntu-24.04 rather than ubuntu-latest, and that is load-bearing. glibc symbol versioning is forward-compatible only, so building against a newer glibc than the base links fine and dies at exec time. cc-debian13 is 2.41, which makes 24.04 (2.39) the newest usable runner — 25.10 is 2.42 and 26.04 is 2.43. A floating ubuntu-latest would break the image the day it rolls over. `docker build .` still works from a clean clone. The Dockerfile declares a global ARG BIN_SOURCE=source and selects between a source-compiled stage and a copy-the-artifact stage; BuildKit only builds the stage that resolves, so CI's --build-arg BIN_SOURCE=prebuilt never touches the Rust builder. The ARG must sit before the first FROM to be usable in one, and the selection has to go through a FROM — --from takes no variable expansion. Dropped cache-from/cache-to on the Docker build: nothing compiles in the image now, and mode=max was stuffing Rust build layers into the same repo-wide 10GB budget Swatinem/rust-cache uses, so the two evicted each other. The compile jobs now save-if main, since branch-scoped caches aren't readable from other branches and PR runs were only evicting the entry they restore from. Images build on pull requests too. Nothing else builds the image, so a broken Dockerfile only surfaced after merge. Adds linux-aarch64 to the release assets, which the arm64 image needed anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
The container image is distroless as of this version and no longer has a shell, which is a visible change for anyone who execs into it. Release assets gain linux-aarch64, and the Linux binaries are built against an older glibc than before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #53. Same shape as radiosilence/nano-web#21, adapted to the fact that this binary can't be static. Also bumps to v3.3.1, so merging this cuts a release and exercises the new publish path end to end.
What
The image build compiled the whole crate a second time inside Docker and was the slowest thing in the pipeline:
With
lto = trueandcodegen-units = 1those are the most expensive compiles in the repo, and we paid for two of them twice per push.Load-bearing decisions
One job, one compile, two outputs.
build-imagecompiles the Linux target, copies the binary todist/, uploads it as the release asset, and feeds the same file to the Docker build. A separate build job would cost a runner handoff and an artifact round-trip, andupload-artifactdrops the executable bit. One job makes both problems disappear.Distroless runtime. The binary is the only thing the image needs to contain — no shell, no package manager, no apt layer. It's
gcr.io/distroless/cc-debian13rather thanscratchbecause three things rule scratch out, each verified against the actual image:/etc/ssl/certs/ca-certificates.crtreqwestresolves roots viarustls-native-certs, which reads the system trust storelibstdc++/libgcc_s/ loaderkreuzbergembeds pdfium and dlopens it at runtime — also why static musl is out/tmpstd::env::temp_dir())This is a visible change for anyone exec'ing into the container — noted in the changelog, with the
:debugtag as the escape hatch.Runners are pinned, and that's load-bearing. glibc symbol versioning is forward-compatible only, so building against a newer glibc than the base links fine and dies at exec time. Measured:
debian:bookworm-slim(source-path builder)ubuntu-24.04(CI runner)distroless/cc-debian13(base)ubuntu:25.10ubuntu:26.04A floating
ubuntu-latestwould break the image the day it rolls over, so it'subuntu-24.04/ubuntu-24.04-armexplicitly. (cc-debian12is 2.36 and wouldn't clear the runners at all.) #55 tracks decoupling this properly.docker build .still works from a clean clone. A globalARG BIN_SOURCE=sourceselects between a source-compiled stage and a copy-the-artifact stage; BuildKit only builds the stage that resolves, so CI's--build-arg BIN_SOURCE=prebuiltnever touches the Rust builder. Two constraints found the hard way: theARGmust precede the firstFROMto be usable in one, and the selection must route through aFROM—COPY --from=rejects variable expansion outright (variable expansion is not supported for --from).Caching
cache-from/cache-to: type=gha,mode=maxon the Docker build. Nothing compiles in the image now, andmode=maxwas stuffing Rust build layers into the same repo-wide 10GB budgetSwatinem/rust-cacheuses — the two evicted each other.save-if: github.ref == 'refs/heads/main'. Branch-scoped caches aren't readable from other branches, so PR runs were writing entries whose only effect was evicting themainentry they restore from.Other
docker run --rm <image> --versionsmoke test per arch.linux-aarch64to the release assets, which the arm64 image needed anyway.Verifying
Locally, both Dockerfile paths:
--build-arg BIN_SOURCE=prebuiltin a context with noCargo.toml→ succeeds and never pullsrust:1-bookworm, confirming the source stage is pruned.dist/references in the build log, proceeds into thebuilderstage, full build exits 0.On merge,
Publishruns for real: pushes both arch images, creates the:main/:sha-/semver manifests, then cuts v3.3.1 last. Worth watching, since it's the first exercise of the load-and-push path and of the release-gating-on-latest-release logic from #53.The
--versionsmoke test is now the only thing between a broken image and the registry, and there's no shell in there to debug with afterwards.🤖 Generated with Claude Code
https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq