Build fails with `sh: 1: cargo: not found`

[kariae]

Maintainer's Note: This has become a meta-issue of multiple bugs all causing the host toolchain to not be mounted in the container, and includes:

• Docker in Docker (Docker in docker #387)
• Nix Store (docker: Add Nix Store volume support #524)
• Remote Docker support (Recommended GitLab usage #273, Add comprehensive support for remote docker. #785)
• WSL2
• WSL2 Docker-in-Docker (Cannot run cross in Docker in Docker in WSL #728)
• Running cross in parallel (Error when building in parallel: sh: 1: cargo: not found #496)
• Docker-in-Docker Fails to mount host dirs (Docker in Docker doesn't seem to find / forward the volume mounts #1283)

---

The build fails on a Docker container with sh: 1: cargo: not found, the Docker container is run with a custom image that contains Docker with a wrapper to have DinD working, and Rust nightly docker run -it -v pwd:/var/tmp -v /var/run/docker.sock:/var/run/docker.sock kariae/docker-rust-nightly sh the command fails with sh: 1: cargo: not found

[evanjs]

I am also encountering this issue.
Nothing special about my setup, though.

[evanjs@nixentoo:~/src/scratch/sdl2test]$ cross build --target x86_64-pc-windows-gnu
Unable to find image 'japaric/x86_64-pc-windows-gnu:v0.1.14' locally
v0.1.14: Pulling from japaric/x86_64-pc-windows-gnu
b8a43e1b7983: Pull complete 
5c5185a06822: Pull complete 
8458c4a4e317: Pull complete 
6d0388b275d4: Pull complete 
56603a33e04b: Pull complete 
2cedd67f6904: Pull complete 
5d40de413ec5: Pull complete 
fbd3dda3b3f5: Pull complete 
ee23f3a81425: Pull complete 
22d70475d56a: Pull complete 
cc2c4c043bf3: Pull complete 
ace8770c3426: Pull complete 
Digest: sha256:b164f497e18f89133de814898d838603b07a6b1feb80b06271ddb3fb86128cc8
Status: Downloaded newer image for japaric/x86_64-pc-windows-gnu:v0.1.14
sh: 1: cargo: not found

Using NixOS 19.03 on 4.19.24
Docker: 18.09.2
Cross: 0.1.14
cargo 1.34.0-nightly (5c6aa46e6 2019-02-22)

[kariae]

I checked the image and I didn't find the installation of cargo which is a little bit tricky, since we use it to build the binaries ?!

[afiore]

I am encountering the same problem. I have just tried running cross build --target aarch64-unknown-linux-musl and got the following output

Unable to find image 'japaric/x86_64-unknown-linux-musl:v0.1.14' locally
v0.1.14: Pulling from japaric/x86_64-unknown-linux-musl
8d51e945c435: Pull complete
d8d21cdaac84: Pull complete
99b5ff394311: Pull complete
3ab3190bc52d: Pull complete
8f2bd7981235: Pull complete
ba0de5bbf3ef: Pull complete
b8380c29d163: Pull complete
e75a875c5cd4: Pull complete
e10da4c5e540: Pull complete
26440e989b5e: Pull complete
b87e83e92416: Pull complete
2e920f245507: Pull complete
Digest: sha256:524b2d33f250ce4e243397b26aab5be7e90d5413b047cc1b9461a62b42b603e6
Status: Downloaded newer image for japaric/x86_64-unknown-linux-musl:v0.1.14
sh: 1: cargo: not found

like @evanjs I am also running on NixOS (18.03.133402.cb0e20d6db9 (Impala))

[wucke13]

I'm also hit by

sh: 1: cargo: not found
failed to resize tty, using default size
sh: 1: cargo: not found

On Linux, NixOS 19.03

[binarylogic]

Same problem here. It appears to be limited to executing cross within a docker image. I'm curious if anyone has workarounds?

[AluisioASG]

The problem is that the image uses the host's own Rust install, which in some cases (e.g. NixOS) cannot be run on Ubuntu. You can work around it by building an image with its own Rust install, for example:

FROM japaric/x86_64-unknown-freebsd:v0.1.4
RUN apt-get update && apt-get install -y wget
RUN mkdir -m777 /opt/rust /opt/cargo
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN wget --https-only --secure-protocol=TLSv1_2 -O- https://sh.rustup.rs | sh /dev/stdin -y
RUN rustup target add x86_64-unknown-freebsd
RUN printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh
RUN chmod +x /usr/local/bin/sh

Rust is installed outside ~ so the user the build runs as can access it. About the last two lines, I wasn't able to override CARGO_HOME from the shell profile so I overrode the shell instead.
Not pretty, but gets the job done.

[wucke13]

Using something from the host breakes many of the otherwise strong guarantees alike "runs fine with me, will run fine with you" docker has to offer. IMO this should be at least considered to be changed.

[reitermarkus]

Have you tried installing cross from master?

[ndusart]

I had still to make some workaround for using cross in gitlab-ci even when using the master branch.
But I managed to do it, you can find information in this comment: #273 (comment)

This probably does not apply for nixOS users. But for the OP surely :)

[zonyitoo]

Same issue was found on circle-ci build: https://circleci.com/gh/shadowsocks/shadowsocks-rust/5

Could that be fixed by your script, too? @AluisioASG

[tomscript]

# cargo install --git https://github.com/rust-embedded/cross.git
    Updating git repository `https://github.com/rust-embedded/cross.git`
  Installing cross v0.2.0-alpha.1 (https://github.com/rust-embedded/cross.git#2f284acd)
<removed>
    Replaced package `cross v0.1.16` with `cross v0.2.0-alpha.1 (https://github.com/rust-embedded/cross.git#2f284acd)` (executable `cross`)

# cross build --target x86_64-pc-windows-gnu
Unable to find image 'rustembedded/cross:x86_64-pc-windows-gnu' locally
x86_64-pc-windows-gnu: Pulling from rustembedded/cross
<removed>
Digest: sha256:cdc542fe0a4b5cf89817388bcb4769f430a2979281add6f5e8fb26baf9f94a9a
Status: Downloaded newer image for rustembedded/cross:x86_64-pc-windows-gnu
sh: 1: cargo: not found

I tried to install cargo from master but running into the same issue. I'm using WSL. Any suggestions on how to troubleshoot further?

[urso]

Did run into a similar issue with development containers and Docker in Docker setup. I created a small POC (PR #387) that tries to find the corrects paths on the host machine when spinning up the child container.

[edenreich]

I can confirm this issue, it also happens on armv7-unknown-linux-musleabihf-0.2.0

shouldn't rust be installed in those container images ? or I misunderstood the way this tool works

FROM rustembedded/cross:armv7-unknown-linux-musleabihf-0.2.0

ENV PATH="/root/.cargo/bin:$PATH"

RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable

[balboah]

same problem, for some reason it worked a while ago. but not now (macOS)

[brainstorm]

I can confirm as well, started happening a couple of days ago. Here I'm not even using an official rustembedded docker image, but a relatively simple Alpine linux one (see rust-bio/rust-htslib#193 if you need more details to reproduce and/or context):

$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/limsadmin/.rustup

installed targets for active toolchain
--------------------------------------

x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.42.0 (b8cedc004 2020-03-09)

$ cat Cross.toml
[target.x86_64-unknown-linux-musl]
image = "brainstorm/rust-htslib-musl-alpine:latest"
$ cat docker/Dockerfile.alpine
FROM alpine:edge

RUN apk add --update wget git g++ build-base automake autoconf openssl openssl-dev curl-dev bzip2-dev xz-dev zlib-dev rustup && \
    rm -rf /var/cache/apk/*

WORKDIR /root
RUN rustup-init -y

ENV PATH "/root/.cargo/bin:$PATH"
RUN rustup target add x86_64-unknown-linux-musl
$ cross build --target x86_64-unknown-linux-musl
sh: cargo: not found