Tracking: RISC-V support in Eth Docker

[lazyprogrammerio]

As the upstream software RISC-V support has come a long way, there is the real possibility of a dockerized approach to Ethereum staking on RISC-V boards like Sifive, Milk-V, LicheePI.

[lazyprogrammerio]

Current state of building and running ETH execution clients on RISC-V:

• geth - WORKS

[lazyprogrammerio]

Current state of building and running ETH consensus clients on RISC-V:

• teku - DOES NOT WORK. building works. Running does not work because of upstream issues with https://github.com/supranational/blst [FIXED by building liblst.so manually] and ROCKSDB https://github.com/facebook/rocksdb [to be investigated]
• lodestar - DOES NOT WORK. building does not work because of upstream issues with Electron https://github.com/electron/electron. https://github.com/riscv-forks/electron might be an alternative, to be investigated, see: How do I consume the releases? riscv-forks/electron-riscv-releases#1
• lighthouse - DOES NOT WORK. building fails as crate https://github.com/RustCrypto/utils/tree/master/cpufeatures does not support RISC-V.
• prysm - DOES NOT WORK. building cannot be started as bazel/bazelisk https://github.com/bazelbuild/bazelisk does not support RISC-V. To be investigated: https://bazel.build/install/compile-source#bootstrap-bazel

[lazyprogrammerio]

For Prysm -> Bazel issue created: bazelbuild/bazel#23018
For Lodestar -> riscv-forks/electron-riscv-releases#1

[lazyprogrammerio]

Lodestar: After deciding to not install the binaries for the Electron dependency to see how far I can go, I reached the big blocker, which is classic-level (a LevelDb wrapper) - no support for RISC-V in the assembly code.

PR was created to see if it fixes the issue: Level/classic-level#94.

One note here, Lodestar depends on a classic-level, which depends on leveldb. Leveldb version used on classic-level seems to be 7 years old (if I am not wrong): https://github.com/google/leveldb/commits/v1.20, see https://github.com/Level/classic-level/commits/main/deps/leveldb.

[haurog]

Nimbus - DOES NOT WORK: gcc: error: ‘-march=native’: ISA string must begin with rv32 or rv64 Probably easy to fix and get a step further by adding rv64 target

Grandine - DOES NOT WORK: error: failed to run custom build command for ring v0.16.20 Seems to fail due to missing arch info in that crate.

[lazyprogrammerio]

Erigon - DOES NOT WORK: github.com/prysmaticlabs/gohashtree@v0.0.3-alpha.0.20230502123415-aafd8b3ca202/hash.go:77:5: undefined: supportedCPU. From how it looks like, the supported amd64 and arm64 arches have the code written in assembly, which is going to be quite a challenge for RISC-V: https://github.com/prysmaticlabs/gohashtree/blob/main/hash_arm64.s

[lazyprogrammerio]

With @haurog's fix, here is a Dockerfile to build Nimbus beacon/validator:

FROM alpine:edge

RUN apk update
RUN apk add nim
RUN nim --version

RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git bash git-lfs

WORKDIR /usr/src

RUN bash -c "git clone --recurse-submodules -j8 https://github.com/lazyprogrammerio/nimbus-eth2 nimbus-eth2"

RUN bash -c "cd nimbus-eth2 && make USE_SYSTEM_NIM=1 -j$(nproc) update"

RUN bash -c "cd nimbus-eth2 && make USE_SYSTEM_NIM=1 -j$(nproc) nimbus_beacon_node nimbus_validator_client"

Output:

Build completed successfully: build/nimbus_validator_client
Build completed successfully: build/nimbus_beacon_node

[lazyprogrammerio]

Docker file to build Nimbus for eth-docker

# Build Nimbus in a stock alpine container
FROM nimbus/devel:stage1 AS builder
# nimbus/devel:stage1 is the above comment Dockerfile built image

# Included here to avoid build-time complaints
ARG DOCKER_TAG
ARG DOCKER_VC_TAG
ARG DOCKER_REPO
ARG DOCKER_VC_REPO

ARG BUILD_TARGET
ARG SRC_REPO

# Pull all binaries into a second stage deploy debian container
FROM alpine:edge AS consensus

ARG USER=user
ARG UID=10002

RUN apk update && apk add  ca-certificates bash tzdata git curl
RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git bash git-lfs
# See https://stackoverflow.com/a/55757473/12429735RUN
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/usr/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"

RUN mkdir -p /var/lib/nimbus/ee-secret && chown -R ${USER}:${USER} /var/lib/nimbus && chmod 700 /var/lib/nimbus && chmod 777 /var/lib/nimbus/ee-secret

# Cannot assume buildkit, hence no chmod
COPY --from=builder --chown=${USER}:${USER} /usr/src/nimbus-eth2/build/nimbus_beacon_node /usr/local/bin/
COPY --chown=${USER}:${USER} ./docker-entrypoint.sh /usr/local/bin/
COPY --chown=${USER}:${USER} ./validator-exit.sh /usr/local/bin/
# Belt and suspenders
RUN chmod -R 755 /usr/local/bin/*

USER ${USER}

ENTRYPOINT ["nimbus_beacon_node"]

FROM alpine:edge AS validator

ARG USER=user
ARG UID=10000

RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git bash git-lfs

# See https://stackoverflow.com/a/55757473/12429735RUN
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/usr/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"

RUN mkdir -p /var/lib/nimbus/ee-secret && chown -R ${USER}:${USER} /var/lib/nimbus && chmod 700 /var/lib/nimbus && chmod 777 /var/lib/nimbus/ee-secret

# Cannot assume buildkit, hence no chmod
COPY --from=builder --chown=${USER}:${USER} /usr/src/nimbus-eth2/build/nimbus_beacon_node /usr/local/bin/
COPY --chown=${USER}:${USER} ./docker-entrypoint.sh /usr/local/bin/
COPY --chown=${USER}:${USER} ./validator-exit.sh /usr/local/bin/
# Belt and suspenders
RUN chmod -R 755 /usr/local/bin/*

USER ${USER}

ENTRYPOINT ["nimbus_beacon_node"]

FROM alpine:edge AS validator

ARG USER=user
ARG UID=10000

RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git bash git-lfs
# See https://stackoverflow.com/a/55757473/12429735RUN
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"

# Create data mount point with permissions
RUN mkdir -p /var/lib/nimbus && chown -R ${USER}:${USER} /var/lib/nimbus && chmod -R 700 /var/lib/nimbus

# Cannot assume buildkit, hence no chmod
COPY --from=builder --chown=${USER}:${USER} /usr/src/nimbus-eth2/build/nimbus_validator_client /usr/local/bin/
COPY --chown=${USER}:${USER} ./docker-entrypoint-vc.sh /usr/local/bin/
# Belt and suspenders
RUN chmod -R 755 /usr/local/bin/*

USER ${USER}

ENTRYPOINT ["nimbus_validator_client"]

[yorickdowne]

I've adjusted Dockerfile.source for Nimbus to build on alpine:3. Test that on your RISC-V machine, if you would.

Can you also get me output of uname -a on that machine, please. That should allow me to adjust ./ethd config so it offers a Nimbus/Geth combo on Risc-V

[yorickdowne]

Went for it by looking for riscv. Try an ./ethd config and see how it behaves for you, please.

[yorickdowne]

You'd need to adjust NIM_SRC_REPO and/or NIM_SRC_BUILD_TARGET manually in .env until that build fix makes it into a release

[haurog]

Here is the output from uname -a
Linux 5.10.113+ #1 SMP PREEMPT Thu Apr 25 13:17:48 UTC 2024 riscv64 riscv64 riscv64 GNU/Linux

[yorickdowne]

Thanks. I could adjust the grep to riscv64 as other riscv architectures won’t work with existing docker images. But, no rush. Let’s see whether it works at all, first

[yorickdowne]

Checking available products. Milk-V Jupiter seems likely with NVMe and 16 GiB RAM. Ditto Sifive HiFive Unmatched Rev. B

Still hard mode compared to Odroid H4 (Ultra), but doable.

LicheePi has no NVMe from what I can see, not a good choice.

[haurog]

That pretty much is our conclusion as well. The boards we have access to at the moment are rather to see if we actually get anything running. Maybe there is a chance to split consensus and execution to two boards, but it still is gonna be difficult. Unfortunately the Jupiter is not widely available at the moment and there are only some preorder units that have been sent out. The development of new CPUs and boards seem to be very fast and in a year or so we might be in a very different position. Ideally we would have clients ready to be used by then.