Multiarchitecture Multistage Rust Docker Builds for Umbrel Apps

Learn to package your Rust code and cryptographic bitcoin magic with Docker for distribution via Umbrel

To create an Umbrel app (or any similar Rapsberry Pi app), you will need to create statically linked binaries contained in docker images that are created from multistage builds, for both x86 and arm targets 💣

You may want to know wtf that all means if you want to ship your Rust code. 🦀

let Docker = "Lego, but not as fun";

Docker is containerization software, best demonstrating it's elegance through the isolation of individual applications into their own sandboxed environment. Docker turns software into lego blocks.

are you feeling it now mr krabs (ﾟ⚙͠ | ∀ |⚙͠)

In the past some of us have hated docker for making software lego, running up against the pains of interacting with and modifying containers software. But slowly, we are converted for reasons:

Umbrel is a bitcoin node-in-a-box platform, and the Umbrel app store is a stylish demonstration of containerisation. Each individual Umbrel application runs within it's own docker container. Programs are isolated into their own long-lived storage file systems, and internet traffic is only allowed to flow through explicitly defined ports.

This standardization and containerization is a superpower for building modular systems that are robust to failure of individual software applications.

Creating a Docker container involves building your code from a Dockerfile, where you specify how to install your software from scratch using a base image. We can think of base images as a fresh linux installation with no deviation such that the installation steps are deterministically guaranteed to work; you can tell me if that is not accurate.

With docker you can have even slimmer images than base linux, such as just using python or the super minimal scratch (virtually empty).

Umbrel ☂

For Umbrel, our bitcoin node (bitcoind) and lightning node (lnd) run in separate containers. If your LND container were to catastrophically fail and crash, then your bitcoind container would be completely unaffected.

Given Umbrel is such a user friendly and approachable node box, and has such a large userbase, Umbrel is primed to distribute your Bitcoin software to the masses.

Rust 🦀

We will refrain from our own evangelism, but Rust is undeniably the rising star of programming languages in Bitcoin and beyond. We are seeing powerful Rust libraries like Bitcoin Dev Kit (BDK) and Lightning Dev Kit (LDK) being built upon, and cryptographic munitions are slowly being pushed out to the people.

In my view there is a significant gap between the vast magic occurring in Bitcoin Rust git repositories, compared to the select few Rust programs that are actually reaching Bitcoiners.

Rust is a compiled language, where a compiler translates the program's source code into machine code that the computer can understand. This step of compiling allows software to be compiled into small packaged executable binaries, which run quickly and can be easily distributed.

However your software dependencies and resulting binary must correspond to a language your machine can understand - the instruction set architecture or computer architecture:

nerd words

Architecture dependence can be annoying, since we need our application to easily work across a range of devices. Umbrel for example, requires x86 and arm support (e.g for your laptop test-env (AMD) and your raspberry-pi prod (ARM)).

How are we to handle this variation of architecture in our clean docker builds?

The Docker File: Multistage Rust x86 and ARM

Suppose

Given a programming language which does not natively implement If-Else statements, there exists a stackoverflow Q&A of how to hack them in. [1]

Here we present to you a Dockerfile that will:

1. Conditionally compile to either x86 or arm, depending on the provided docker build --platform.
2. Utilize multistage builds, meaning that your code is first compiled into a binary, and then just the binary alone is copied into the final docker image. We delete all the build files and dependencies!
3. Use statically linked musl targets of x86_64-unknown-linux-musl and aarch64-unknown-linux-musl, meaning all your dependencies are compiled into the executable, and you hopefully don't have to worry about a thing.

In your cargo repository write a Dockerfile:

# Multistage Rust Docker Build for Umbrel App
# by utxo.club
#
# x86_64-unknown-linux-musl
# aarch64-unknown-linux-musl
#
# Conditionally `cargo build` for platforms of x86_64 or ARM.
# Use musl for static linking, producing a standalone executable with no dependencies.
# In the final Docker stage we copy the built binary to alpine, and run with environment:
# $LND_HOST, $LND_GRPC_PORT, $TLS_FILE, $MACAROON_FILE"

## Initial build Stage 
FROM rustlang/rust:nightly AS builder
# Target architecture argument used to change build
ARG TARGETARCH
# Some nicer rust debugging
ENV RUSTFLAGS="-Z macro-backtrace"
ENV RUST_BACKTRACE=1
# Copy the required build files. In this case, these are all the files that
# are used for both architectures.
WORKDIR /usr/src/loin
COPY Cargo.toml Cargo.lock build.rs ./
COPY src ./src

## x86_64
FROM builder AS branch-version-amd64
RUN echo "Preparing to cargo build for x86_64 (${TARGETARCH})"
# Install the required dependencies to build for `musl` static linking
RUN apt-get update && apt-get install -y musl-tools musl-dev
# Add our x86 target to rust, then compile and install
RUN rustup target add x86_64-unknown-linux-musl
RUN cargo install --features=test_paths --target x86_64-unknown-linux-musl --path .

# ARM
FROM builder AS branch-version-arm64
RUN echo "Preparing to cargo build for arm (${TARGETARCH})"
# Install the required dependencies to build for `musl` static linking for arm.
RUN apt-get update && apt-get install musl-tools clang llvm -y
# Add our arm target to rust, some build variables, then compile and install
RUN rustup target add aarch64-unknown-linux-musl
ENV CC_aarch64_unknown_linux_musl=clang
ENV AR_aarch64_unknown_linux_musl=llvm-ar
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-Clink-self-contained=yes -Clinker=rust-lld"
RUN cargo install --features=test_paths --target aarch64-unknown-linux-musl --path .

# We build for either x86_64 or ARM from above options using the docker $TARGETARCH
FROM branch-version-${TARGETARCH} AS chosen_builder
RUN echo "Called build!"

# Run Loin from a final debian container
FROM debian:buster-slim
# Copy just the binary from our build stage
COPY --from=chosen_builder /usr/local/cargo/bin/loin /usr/local/bin/loin
# Expose any necessary ports
EXPOSE 4444
# Run
CMD ["loin"]

Build and push to Dockerhub with:

sudo docker buildx build --platform linux/arm64,linux/amd64 --tag dockeruser/myapp:0.0.1 --output type=registry .

hellish

How does it work?

Thankfully Docker provides an environment variable ${TARGETARCH} which is set to the the target architecture for our build.

In one build stages, we expand ${TARGETARCH} to conditionally select to build from one of either branch-version-amd64 or branch-version-arm64 (for x86, ARM).

We follow the build path for the relevant architecture using the Dockerfile line:

FROM branch-version-${TARGETARCH} AS chosen_builder

So?

You should build a docker image for your bitcoin Rust project and smoothly push it out to thousands of people, if you don't have any rust bitcoin project you should start one today.

Follow this tutorial to test and release your software on your Umbrel or similar platform like sovereign computing @ Start9.

We have a number of software libraries which we look forward to making front-ends for in the near future, and to then push them out for experimental and distributed use.

Raw Dockerfile

We created this docker image for the Legends of Lightning Hackathon. check out our project nolooking.