Skip to content

fix(orchestrator): create /dev/fuse for dockerd; runner parity refinements - #152

Closed
Bnjoroge1 wants to merge 1 commit into
fix/conformance-campaign-server-and-runtime-fixesfrom
fix/docker-fuse-device-node-and-runner-parity
Closed

fix(orchestrator): create /dev/fuse for dockerd; runner parity refinements#152
Bnjoroge1 wants to merge 1 commit into
fix/conformance-campaign-server-and-runtime-fixesfrom
fix/docker-fuse-device-node-and-runner-parity

Conversation

@Bnjoroge1

@Bnjoroge1 Bnjoroge1 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Follow-ups to the conformance campaign fixes (PR #151).

  • docker: the krunfw guest kernel has fuse built in, but /dev boots as a plain tmpfs with no device nodes, so fuse-overlayfs (dockerd's fallback when its overlay probe fails) dies with fuse: device not found. The hook now creates /dev/fuse when the kernel lists fuse, letting dockerd auto-pick fuse-overlayfs (CoW) — on this kernel dockerd's overlay2 probe mount gets EINVAL and overlay2 is never viable, so the earlier fix was falling back to vfs. vfs is now forced only when overlay fails AND fuse is absent.
  • RLIMIT_NOFILE: 524288 instead of 1048576 — systemd's built-in hard default, which is what GitHub's runner service inherits (exact parity).
  • guest PATH: cargo bin dir matches the runner user (/home/<user>/.cargo) instead of hardcoded /root/.cargo/bin, which the unprivileged runner cannot stat (nodejs/ci EACCES).
  • multiarch shim: add libsystemd0:amd64 (valkey's x86_64 tarballs link libsystemd.so.0).

Verified live on the golden VM: hook creates /dev/fuse, dockerd 28.0.4 reports Storage Driver: fuse-overlayfs, and hello-world runs. 61 orchestrator tests pass.


Summary by cubic

Creates /dev/fuse in krun guests so dockerd can auto-select fuse-overlayfs and keep copy-on-write. Previously we forced vfs when the overlay2 probe failed; now we only force vfs if overlay is unusable and /dev/fuse is missing, and we write daemon.json only when a driver is forced.

  • Create /dev/fuse when the kernel supports fuse; leave DRIVER unset when the overlay2 probe succeeds or fails but fuse exists; force DRIVER=vfs only when the probe fails and fuse is absent.
  • Migration: if you rely on a custom Docker data-root without forcing a driver, it will no longer be written; configure it elsewhere.
  • Runner parity: set RLIMIT_NOFILE=524288, add /home/<user>/.cargo to PATH, and install libsystemd0:amd64 for the multiarch shim.

Written for commit 64d0055. Summary will update on new commits.

Review in cubic

Note

Create /dev/fuse for dockerd and stop forcing overlay2/vfs storage drivers

  • Creates /dev/fuse (via mknod c 10 229) when /proc/filesystems reports fuse support and the node is missing, in docker_start_command in lib.rs
  • On successful overlay probe, no longer forces overlay2; leaves DRIVER empty so dockerd selects its own driver
  • On failed overlay probe, only forces vfs when /dev/fuse is absent; otherwise leaves DRIVER empty to allow dockerd's fallback
  • Writes /etc/docker/daemon.json (including data-root and storage-driver) only when DRIVER is non-empty
  • Risk: data-root is no longer written to daemon.json when no explicit driver is selected; any setup relying on a custom data-root without a forced driver will lose that configuration
📊 Macroscope summarized 64d0055. 1 file reviewed, 2 issues evaluated, 0 issues filtered, 1 comment posted

🗂️ Filtered Issues

@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 365c42d2-b207-4ad3-95c1-224dc1a0bfac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@Bnjoroge1

Copy link
Copy Markdown
Collaborator Author

@macroscope review

@macroscopeapp

macroscopeapp Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Manual reviews triggered for commit 2392540:

All prior checks · these links stay valid even if you push more commits.

@macroscopeapp

macroscopeapp Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Just FYI for future @mentions, I'm Macroscope-App, not Macroscope.

Review triggered. The Approvability and Correctness checks are running now and will post results when complete.

@Bnjoroge1
Bnjoroge1 marked this pull request as ready for review August 20, 2026 02:05
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@Bnjoroge1
Bnjoroge1 force-pushed the fix/docker-fuse-device-node-and-runner-parity branch from 2392540 to c5533c8 Compare August 20, 2026 03:07
Comment on lines +1111 to +1113
if [ -n \"$DRIVER\" ]; then \
printf '{{\"data-root\":\"{DOCKER_DATA_ROOT}\",\"storage-driver\":\"%s\"}}\\n' \"$DRIVER\" > /etc/docker/daemon.json; \
fi; \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High src/lib.rs:1111

When DRIVER is empty, this leaves stale /etc/docker/daemon.json untouched or creates no config, so Docker can retain an unusable storage-driver and use /var/lib/docker instead of /storage/docker. Write data-root unconditionally, and include storage-driver only when DRIVER is forced.

-             if [ -n "$DRIVER" ]; then \
-               printf '{{"data-root":"{DOCKER_DATA_ROOT}","storage-driver":"%s"}}\\n' "$DRIVER" > /etc/docker/daemon.json; \
-             fi; \
+             if [ -n "$DRIVER" ]; then \
+               printf '{{"data-root":"{DOCKER_DATA_ROOT}","storage-driver":"%s"}}\\n' "$DRIVER" > /etc/docker/daemon.json; \
+             else \
+               printf '{{"data-root":"{DOCKER_DATA_ROOT}"}}\\n' > /etc/docker/daemon.json; \
+             fi; \
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @crates/preloop-orchestrator/src/lib.rs around lines 1111-1113:

When `DRIVER` is empty, this leaves stale `/etc/docker/daemon.json` untouched or creates no config, so Docker can retain an unusable `storage-driver` and use `/var/lib/docker` instead of `/storage/docker`. Write `data-root` unconditionally, and include `storage-driver` only when `DRIVER` is forced.

@Bnjoroge1
Bnjoroge1 force-pushed the fix/conformance-campaign-server-and-runtime-fixes branch from 94c62db to d0215e2 Compare August 20, 2026 03:10
…ments

Follow-ups to the conformance campaign fixes (PR #151):

- docker: the krunfw guest kernel has fuse built in, but /dev boots as a
  plain tmpfs with no device nodes, so fuse-overlayfs (dockerd's fallback
  when its overlay probe fails) dies with 'fuse: device not found'. The
  hook now creates /dev/fuse when the kernel lists fuse, letting dockerd
  auto-pick fuse-overlayfs (CoW) — on this kernel dockerd's overlay2 probe
  mount gets EINVAL and overlay2 is never viable, so the earlier fix was
  falling back to vfs. vfs is now forced only when overlay fails AND fuse
  is absent.
- RLIMIT_NOFILE: 524288 instead of 1048576 — systemd's built-in hard
  default, which is what GitHub's runner service inherits (exact parity).
- guest PATH: cargo bin dir matches the runner user (/home/<user>/.cargo)
  instead of hardcoded /root/.cargo/bin, which the unprivileged runner
  cannot stat (nodejs/ci EACCES).
- multiarch shim: add libsystemd0:amd64 (valkey's x86_64 tarballs link
  libsystemd.so.0).

Verified live on the golden VM: hook creates /dev/fuse, dockerd 28.0.4
reports Storage Driver: fuse-overlayfs, and hello-world runs. 61
orchestrator tests pass.
@Bnjoroge1
Bnjoroge1 force-pushed the fix/docker-fuse-device-node-and-runner-parity branch from c5533c8 to 64d0055 Compare August 20, 2026 03:11
@Bnjoroge1
Bnjoroge1 deleted the branch fix/conformance-campaign-server-and-runtime-fixes August 20, 2026 03:17
@Bnjoroge1 Bnjoroge1 closed this Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant