Skip to content

ci: build docker images from prebuilt binaries, on a distroless base - #54

Merged
radiosilence merged 4 commits into
mainfrom
ci/reuse-binaries-in-image
Jul 26, 2026
Merged

ci: build docker images from prebuilt binaries, on a distroless base#54
radiosilence merged 4 commits into
mainfrom
ci/reuse-binaries-in-image

Conversation

@radiosilence

@radiosilence radiosilence commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Follow-up to #53. Same shape as radiosilence/nano-web#21, adapted to the fact that this binary can't be static. Also bumps to v3.3.1, so merging this cuts a release and exercises the new publish path end to end.

What

The image build compiled the whole crate a second time inside Docker and was the slowest thing in the pipeline:

job before
Build arm64 Image 677s
Build amd64 Image 530s
Build linux-x86_64 (same compile) 345s

With lto = true and codegen-units = 1 those are the most expensive compiles in the repo, and we paid for two of them twice per push.

Load-bearing decisions

One job, one compile, two outputs. build-image compiles the Linux target, copies the binary to dist/, uploads it as the release asset, and feeds the same file to the Docker build. A separate build job would cost a runner handoff and an artifact round-trip, and upload-artifact drops the executable bit. One job makes both problems disappear.

Distroless runtime. The binary is the only thing the image needs to contain — no shell, no package manager, no apt layer. It's gcr.io/distroless/cc-debian13 rather than scratch because three things rule scratch out, each verified against the actual image:

need why
/etc/ssl/certs/ca-certificates.crt reqwest resolves roots via rustls-native-certs, which reads the system trust store
libstdc++ / libgcc_s / loader kreuzberg embeds pdfium and dlopens it at runtime — also why static musl is out
/tmp where pdfium is extracted on first use (std::env::temp_dir())

This is a visible change for anyone exec'ing into the container — noted in the changelog, with the :debug tag as the escape hatch.

Runners are pinned, and that's load-bearing. glibc symbol versioning is forward-compatible only, so building against a newer glibc than the base links fine and dies at exec time. Measured:

image glibc
debian:bookworm-slim (source-path builder) 2.36
ubuntu-24.04 (CI runner) 2.39 ✅ newest usable
distroless/cc-debian13 (base) 2.41 — ceiling
ubuntu:25.10 2.42
ubuntu:26.04 2.43

A floating ubuntu-latest would break the image the day it rolls over, so it's ubuntu-24.04 / ubuntu-24.04-arm explicitly. (cc-debian12 is 2.36 and wouldn't clear the runners at all.) #55 tracks decoupling this properly.

docker build . still works from a clean clone. A global ARG BIN_SOURCE=source selects between a source-compiled stage and a copy-the-artifact stage; BuildKit only builds the stage that resolves, so CI's --build-arg BIN_SOURCE=prebuilt never touches the Rust builder. Two constraints found the hard way: the ARG must precede the first FROM to be usable in one, and the selection must route through a FROMCOPY --from= rejects variable expansion outright (variable expansion is not supported for --from).

Caching

  • Dropped cache-from/cache-to: type=gha,mode=max on the Docker build. Nothing compiles in the image now, and mode=max was stuffing Rust build layers into the same repo-wide 10GB budget Swatinem/rust-cache uses — the two evicted each other.
  • Compile jobs now save-if: github.ref == 'refs/heads/main'. Branch-scoped caches aren't readable from other branches, so PR runs were writing entries whose only effect was evicting the main entry they restore from.

Other

  • Images build on PRs — nothing else builds the image, so a broken Dockerfile only surfaced after merge. Includes a docker run --rm <image> --version smoke test per arch.
  • Darwin binaries stay main-only — no image consumes them.
  • Adds linux-aarch64 to the release assets, which the arm64 image needed anyway.

Verifying

Locally, both Dockerfile paths:

  • --build-arg BIN_SOURCE=prebuilt in a context with no Cargo.toml → succeeds and never pulls rust:1-bookworm, confirming the source stage is pruned.
  • Default → zero dist/ references in the build log, proceeds into the builder stage, full build exits 0.
  • Exec on distroless checked with a real glibc-2.36-linked ELF running on the 2.41 base (a shell script wouldn't prove anything, since there's no shell). Runs clean.

On merge, Publish runs for real: pushes both arch images, creates the :main/:sha-/semver manifests, then cuts v3.3.1 last. Worth watching, since it's the first exercise of the load-and-push path and of the release-gating-on-latest-release logic from #53.

The --version smoke test is now the only thing between a broken image and the registry, and there's no shell in there to debug with afterwards.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq

@radiosilence
radiosilence changed the base branch from ci/pipeline-order-and-gating to main July 26, 2026 19:16
The image builds compiled the whole crate a second time inside Docker and
were the two slowest jobs in the pipeline — 677s for arm64 and 530s for
amd64, against 345s for the same compile in the binary job. They now copy the
binary the release already builds, which drops them to a COPY plus a smoke
test.

That constrains glibc: symbol versioning is forward-compatible only, so a
binary built on ubuntu-latest (2.39) links fine and dies at exec time on
debian:bookworm-slim (2.36). The Linux binaries therefore build inside
rust:1-bookworm rather than on the runner, which also lowers the glibc floor
on the published tarballs.

Static musl would remove the constraint entirely but isn't available here:
kreuzberg embeds pdfium via include_bytes! and extracts and dlopens it at
runtime, which a fully static binary cannot do. ca-certificates stays in the
image for a related reason — reqwest resolves roots through
rustls-native-certs, which reads the system trust store rather than carrying
its own, so a scratch base would fail every handshake. Both constraints are
documented in the Dockerfile.

Images now build on pull requests too. Nothing else in the pipeline builds
the image, so a broken Dockerfile only surfaced after merge; it costs seconds
now that there is no compile in it. The darwin binaries stay main-only since
no image consumes them and macOS runners bill at ten times the Linux rate.

Adds linux-aarch64 to the release assets, which the arm64 image needed
anyway. The cost is that the Dockerfile is no longer standalone-buildable; it
carries a copy-paste one-liner for building it by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
@radiosilence
radiosilence force-pushed the ci/reuse-binaries-in-image branch from 95f07ed to a08f6dd Compare July 26, 2026 19:17
The image build compiled the whole crate a second time inside Docker and was
the slowest thing in the pipeline — 677s for arm64 and 530s for amd64, against
345s for the same compile in the binary job. With lto = true and
codegen-units = 1 those are the most expensive compiles in the repo, and we
paid for two of them twice per push.

build-image now owns that compile outright and copies the result into the
image. Keeping it in one job rather than handing off from a separate build job
avoids a runner handoff and an artifact round-trip, and sidesteps
upload-artifact dropping the executable bit.

ubuntu-22.04 is the glibc floor, not an arbitrary pin: 2.35 against
bookworm-slim's 2.36. Symbol versioning is forward-compatible only, so
building on ubuntu-latest (2.39) would link fine and die at exec time in the
image. It also makes the published tarballs portable further back than they
were.

`docker build .` still works from a clean clone. The Dockerfile declares a
global ARG BIN_SOURCE=source and selects between a source-compiled stage and a
copy-the-artifact stage; BuildKit only builds the stage that resolves, so CI's
--build-arg BIN_SOURCE=prebuilt never touches the Rust builder. The ARG must
sit before the first FROM to be usable in one, and the selection has to go
through a FROM — --from takes no variable expansion.

Static musl would remove the glibc question entirely but isn't available here:
kreuzberg embeds pdfium via include_bytes! and extracts and dlopens it at
runtime, which a fully static binary cannot do. ca-certificates stays for a
related reason — reqwest resolves roots through rustls-native-certs, which
reads the system trust store, so a scratch base would fail every handshake.

Dropped cache-from/cache-to on the Docker build: nothing compiles in the image
now, and mode=max was stuffing Rust build layers into the same repo-wide 10GB
budget Swatinem/rust-cache uses, so the two evicted each other. The compile
jobs now save-if main, since branch-scoped caches aren't readable from other
branches and PR runs were only evicting the entry they restore from.

Images build on pull requests too. Nothing else builds the image, so a broken
Dockerfile only surfaced after merge. Adds linux-aarch64 to the release
assets, which the arm64 image needed anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
@radiosilence radiosilence changed the title ci: build the binary once and copy it into the image ci: build docker images from prebuilt binaries Jul 26, 2026
The image build compiled the whole crate a second time inside Docker and was
the slowest thing in the pipeline — 677s for arm64 and 530s for amd64, against
345s for the same compile in the binary job. With lto = true and
codegen-units = 1 those are the most expensive compiles in the repo, and we
paid for two of them twice per push.

build-image now owns that compile outright and copies the result into the
image. Keeping it in one job rather than handing off from a separate build job
avoids a runner handoff and an artifact round-trip, and sidesteps
upload-artifact dropping the executable bit.

The runtime image is now distroless: the binary is the only thing it needs to
contain, so there is no shell, no package manager and no apt layer. It is not
scratch because three things rule that out, each verified against the image:
ca-certificates.crt, because reqwest resolves roots through
rustls-native-certs and reads the system trust store; libstdc++ and a dynamic
loader, because kreuzberg embeds pdfium and dlopens it at runtime (which is
also why a fully static musl build is out); and /tmp, where pdfium is
extracted on first use.

Runners are pinned to ubuntu-24.04 rather than ubuntu-latest, and that is
load-bearing. glibc symbol versioning is forward-compatible only, so building
against a newer glibc than the base links fine and dies at exec time.
cc-debian13 is 2.41, which makes 24.04 (2.39) the newest usable runner —
25.10 is 2.42 and 26.04 is 2.43. A floating ubuntu-latest would break the
image the day it rolls over.

`docker build .` still works from a clean clone. The Dockerfile declares a
global ARG BIN_SOURCE=source and selects between a source-compiled stage and a
copy-the-artifact stage; BuildKit only builds the stage that resolves, so CI's
--build-arg BIN_SOURCE=prebuilt never touches the Rust builder. The ARG must
sit before the first FROM to be usable in one, and the selection has to go
through a FROM — --from takes no variable expansion.

Dropped cache-from/cache-to on the Docker build: nothing compiles in the image
now, and mode=max was stuffing Rust build layers into the same repo-wide 10GB
budget Swatinem/rust-cache uses, so the two evicted each other. The compile
jobs now save-if main, since branch-scoped caches aren't readable from other
branches and PR runs were only evicting the entry they restore from.

Images build on pull requests too. Nothing else builds the image, so a broken
Dockerfile only surfaced after merge. Adds linux-aarch64 to the release
assets, which the arm64 image needed anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
@radiosilence radiosilence changed the title ci: build docker images from prebuilt binaries ci: build docker images from prebuilt binaries, on a distroless base Jul 26, 2026
The container image is distroless as of this version and no longer has a
shell, which is a visible change for anyone who execs into it. Release assets
gain linux-aarch64, and the Linux binaries are built against an older glibc
than before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHzrrbZuVjY9PsBEnKMcFq
@radiosilence
radiosilence merged commit 731fb73 into main Jul 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant