Add OCI image support

[henrybear327]

Prior work: #34
Tracking issue: #31

We introduce elfuse-oci, a standalone Go companion binary that owns the OCI image pipeline: pull, unpack, inspect, run, list (alias images), rmi, and prune, backed by a real OCI image-layout store. elfuse itself stays a pure Linux syscall-to-Darwin runtime with no OCI awareness; the two binaries meet only at the existing elfuse --sysroot <rootfs> <entrypoint> <args> launch path.

OCI images are used here strictly as a distribution format for Linux root filesystems — a reproducible replacement for hand-built --sysroot trees. This is not a container runtime. There is no isolation layer: the guest shares the host network identity, PID space, and clock, and unresolved guest paths fall back to host truth. A workload that needs namespaces, cgroups, port mapping, exec into a running container, or a daemon needs a real container runtime, not an ELF personality.

Why a separate Go binary

The OCI ecosystem is a Go ecosystem. Rather than grow the C runtime with an image pipeline, the acquisition/lifecycle half is ~4.4k lines of Go (plus ~7.3k lines of tests) built on github.com/google/go-containerregistry, and validated for on-disk conformance against crane, skopeo, and umoci. Keeping it out of the C tree keeps both sides small and lets each lean on its native tooling.

Design

docs/oci-design.md is the source of truth: the C/Go boundary, the image-layout store and its refs.json pin table, hardened layer application, the run paths, the concurrency/locking model, and an explicit accounting of which OCI features are and are not implemented. docs/usage.md covers the commands; docs/testing.md the offline/CI split; docs/internals.md the host-literal path fallback.

Highlights:

• Store: spec-shape OCI image-layout other tools can read; refs pinned by digest; an exclusive flock serializes metadata writes so concurrent pulls cannot lose pins.
• Unpack: os.Root-bounded extraction (no symlink/hardlink escape), correct whiteout/opaque handling, staged temp-dir + atomic rename so readers never see a partial tree.
• run: resolves Entrypoint/Cmd/Env/User/WorkingDir with env(1)/Docker precedence, guarantees a PATH, resolves symbolic --user against the image's own /etc/passwd+/etc/group (no-follow), injects host /etc/{resolv.conf,hosts,hostname}, then execs elfuse in place so signals and the pid pass through.
• macOS rootfs: per-digest case-sensitive APFS sparsebundle with per-run clonefile COW; liveness/lifecycle decided by per-digest advisory flocks (attach.lock/run.lock), not pids. --plain-rootfs remains available.
• Lifecycle GC: rmi/prune use reachability GC (shared blobs survive while any ref reaches them) and never reclaim a cache a live run still holds — the run-lock rides through the exec into elfuse and releases exactly on guest exit.

Try it

make elfuse elfuse-oci

build/elfuse-oci pull alpine:3
build/elfuse-oci inspect alpine:3
build/elfuse-oci list

build/elfuse-oci run alpine:3 /bin/sh -c 'echo hello from elfuse'
build/elfuse-oci run --entrypoint /usr/local/bin/python3 python:3.12 \
  -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'

build/elfuse-oci rmi --force alpine:3
build/elfuse-oci prune --cache --all

Images are stored under $ELFUSE_OCI_STORE, or ~/.local/share/elfuse/oci by default.

CI

Split by runner capability

• a Linux job exercises the pure-Go store paths (pull, unpack, inspect, lifecycle) plus image-layout conformance and cross-tool interop (crane/skopeo/umoci) with no Hypervisor.framework
• a hosted macOS job builds and tests the darwin sparsebundle code and a run-less lifecycle smoke

---

Summary by cubic

Adds OCI image support with a new Go CLI, elfuse-oci, covering pull, unpack, inspect, list, run, remove, and prune on a real OCI image‑layout store. Refactors guest bring‑up into shared elfuse_launch and adds --user, --workdir, and --env for precise runtime control; default macOS runs use case‑sensitive APFS sparsebundles with per‑run COW clones.

• New Features

  • Runtime/launch: extracted shared elfuse_launch; --user, --workdir, --env/--clear-env; initial IDs staged so auxv AT_UID/GID match; forwards INT/TERM/QUIT/HUP; uses -- before the guest argv and execs elfuse --sysroot; host /etc/{resolv.conf,hosts,hostname} injected; envp generalized so the host environ is optional.
  • OCI lifecycle: elfuse-oci (on github.com/google/go-containerregistry) defaults to linux/arm64 with --platform; adds pull, unpack (hardened apply with whiteouts/opaque and staged rename), inspect (--json), list (--json), run, rmi, and prune (--cache, --all, --dry-run). Pins refs by digest with store locking and reachability GC. macOS uses sparsebundles with per‑run COW clones and attach/run locks; stale mounts are detached and .elfuse-keep clones are preserved. Plain rootfs remains available (--plain-rootfs) with a sibling per‑digest lock made exec‑survivable. rmi/prune skip busy caches, refuse live ones, and honor --keep; default PATH and symbolic --user resolution happen inside the image rootfs.

• Docs & CI

  • Docs: added docs/oci-design.md; expanded usage, testing, and internals (host‑literal fallback, PATH defaults); README and Makefile updated to build elfuse-oci.
  • CI: OCI image‑layout conformance and cross‑tool interop (crane, skopeo, umoci) on Linux; macOS darwin build, unit tests, sparsebundle round‑trip, and run‑less lifecycle; self‑hosted end‑to‑end HVF runs plus lifecycle teardown guardrails; per‑image workload smokes (python/node/go/jvm/c) that boot under HVF and exercise characteristic operations; shared setup action with fail‑fast guard on superseded PR commits.

^{Written for commit 031824e. Summary will update on new commits.}

[jserv]

Commands to try out

build/elfuse oci list
build/elfuse oci pull alpine:3
build/elfuse oci inspect alpine:3
build/elfuse oci list
build/elfuse oci rmi --force e7a1a92a5bfe 
build/elfuse oci list
build/elfuse-container run --entrypoint /usr/local/bin/python3 \\n    python:3.12 -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'
build/elfuse oci run alpine:3 /bin/sh -c 'echo hello from elfuse'

Leveraging existing Go-based tools and packages is a great step toward full OCI image support. I agree with this change.

I suggest repositioning the build/elfuse binary as an efficient Linux syscall-to-macOS/Darwin runtime, while implementing build/elfuse-container in Go using OCI-related packages.

In other words, build/elfuse oci should not be considered a valid command. OCI-specific functionality should reside in elfuse-container, not in the elfuse executable.

[Max042004]

unpack fails on any image that ships setuid/setgid files when the host
process is unprivileged, which includes stock Debian-based images:

$ elfuse-container unpack --rootfs /tmp/gcc-rootfs gcc:14
elfuse-container: unpack: layer 0: entry "usr/bin/chage": chmodat usr/bin/chage: operation not permitted

gcc:14, ubuntu:24.04, and anything else carrying the shadow suite
(chage, su, passwd, ...) cannot be unpacked at all; the alpine/slim
images used by the CI smoke tests just happen to contain no setuid entries,
so the path is never exercised there. The run cache fill goes through the
same applyMode, so those images fail to run, too.

Root cause: applyMode applies the tar header mode verbatim via
root.Chmod(name, perm|special). The unpacked tree is owned by the invoking
user rather than the tar's root/shadow owners, and an unprivileged chmod
that sets setuid/setgid on such files is rejected with EPERM on macOS, so
the whole unpack aborts on the first such entry.

Since elfuse remaps uid/gid and does not implement setuid-exec semantics,
the special bits carry no meaning at runtime today. I'd suggest degrading
gracefully instead of failing the unpack — retry with the plain permission
bits, and only for this specific case so genuine chmod failures still
surface

[jserv]

After several major iterations, I am not sure if elfuse-container is a good name, as I was also a bit misled by going down trying to support container execution.

Alternatively, name it elfuse-oci.

[henrybear327]

unpack fails on any image that ships setuid/setgid files when the host process is unprivileged, which includes stock Debian-based images:

$ elfuse-container unpack --rootfs /tmp/gcc-rootfs gcc:14
elfuse-container: unpack: layer 0: entry "usr/bin/chage": chmodat usr/bin/chage: operation not permitted

gcc:14, ubuntu:24.04, and anything else carrying the shadow suite (chage, su, passwd, ...) cannot be unpacked at all; the alpine/slim images used by the CI smoke tests just happen to contain no setuid entries, so the path is never exercised there. The run cache fill goes through the same applyMode, so those images fail to run, too.

Root cause: applyMode applies the tar header mode verbatim via root.Chmod(name, perm|special). The unpacked tree is owned by the invoking user rather than the tar's root/shadow owners, and an unprivileged chmod that sets setuid/setgid on such files is rejected with EPERM on macOS, so the whole unpack aborts on the first such entry.

Since elfuse remaps uid/gid and does not implement setuid-exec semantics, the special bits carry no meaning at runtime today. I'd suggest degrading gracefully instead of failing the unpack — retry with the plain permission bits, and only for this specific case so genuine chmod failures still surface

Thanks for testing and reporting the bug. I will address the issue by emitting an error message such that we are not silently dropping special bits without the user knowing about it.

[henrybear327]

After several major iterations, I am not sure if elfuse-container is a good name, as I was also a bit misled by going down trying to support container execution.

Alternatively, name it elfuse-oci.

Renamed accordingly and updated the PR description.

[henrybear327]

unpack fails on any image that ships setuid/setgid files when the host process is unprivileged, which includes stock Debian-based images:

$ elfuse-container unpack --rootfs /tmp/gcc-rootfs gcc:14
elfuse-container: unpack: layer 0: entry "usr/bin/chage": chmodat usr/bin/chage: operation not permitted

gcc:14, ubuntu:24.04, and anything else carrying the shadow suite (chage, su, passwd, ...) cannot be unpacked at all; the alpine/slim images used by the CI smoke tests just happen to contain no setuid entries, so the path is never exercised there. The run cache fill goes through the same applyMode, so those images fail to run, too.
Root cause: applyMode applies the tar header mode verbatim via root.Chmod(name, perm|special). The unpacked tree is owned by the invoking user rather than the tar's root/shadow owners, and an unprivileged chmod that sets setuid/setgid on such files is rejected with EPERM on macOS, so the whole unpack aborts on the first such entry.
Since elfuse remaps uid/gid and does not implement setuid-exec semantics, the special bits carry no meaning at runtime today. I'd suggest degrading gracefully instead of failing the unpack — retry with the plain permission bits, and only for this specific case so genuine chmod failures still surface

Thanks for testing and reporting the bug. I will address the issue by emitting an error message such that we are not silently dropping special bits without the user knowing about it.

Addressed.

Also added 5 workload images listed in #224 to the CI as a test.

Will make the CI tasks simpler for now, as the Go test currently failed due to it hitting elfuse's fixed thread-table cap, and the python test triggers an unhandled guest CPU exception in elfuse.