Month: January 2022

TL;DR: adding ~21 lines of code can remove the need for pre-compiled NIF files if you want to cross-compile an Elixir library that relies on rustler to your nerves build.

Rustler is a Safe Rust bridge for creating Erlang NIF functions, and Nerves lets you Craft and deploy bulletproof embedded software in Elixir. Now, let's put them together, in a better and easier way.

There is an html5ever_elixir repo on the rusterlium GitHub organisation. It is a great project that allows you to use html5ever in Elixir. It also comes with some pre-compiled NIF files covering the most commonly seen operating systems, CPU architectures and ABIs.

If you want to want to deploy any Elixir library that relies on rustler to your Nerves project, you'll have to somehow obtain a pre-compiled NIF file that corresponds to your Nerves build target. For example, say you want to deploy to a Raspberry Pi 4, then you'll need to pre-compile your library with the target named aarch64-unknown-linux-gnu. You can do that by

• pre-compiling the NIF file on your own machine

• downloading the pre-compiled the NIF file from the library's provider, like html5ever_elixir

Either way, you have to do something manually. How do we avoid those tedious things?

Well, it's not quite hard to smooth out the process as rust can do cross-compile for you as long as you have (1) the target added to your host machine, (2) and the cross-compile toolchain.

The first requirement is super easy to satisfy:

# use aarch64-unknown-linux-gnu here
# you can see the full list by
# rustc --print target-list
rustup target add aarch64-unknown-linux-gnu

As for the second requirement, Nerves will automatically download the pre-built toolchain.

mix nerves.new nerves_project
cd nerves_project
export MIX_TARGET=rpi4
mix deps.get
# the root directory of this toolchain will be
# ~/.nerves/artifacts/nerves_toolchain_\${NERVES_TARGET_TRIPLE}-\${HOST_OS}_\${HOST_ARCH}-\${TOOLCHAIN_VER}/

${NERVES_TARGET_TRIPLE} here will be aarch64_nerves_linux_gnu. ${HOST_OS} is either linux or darwin, depends on which OS you're using. The same goes for ${HOST_ARCH}, depends on your host machine's CPU arch. Let denote the root directory of the toolchain as ${TOOLCHAIN_ROOT} (we will refer to this path later).

To cross-compile on the host, a few lines of code needs to be added to the library's mix.exs file.

Firstly, we can see that ${NERVES_TARGET_TRIPLE} is different from the target triple used in rustc, which is aarch64-unknown-linux-gnu. Therefore we need a compile-time map for the translation

  @nerves_rust_target_triple_mapping %{
    "armv6-nerves-linux-gnueabihf": "arm-unknown-linux-gnueabihf",
    "armv7-nerves-linux-gnueabihf": "armv7-unknown-linux-gnueabihf",
    "aarch64-nerves-linux-gnu": "aarch64-unknown-linux-gnu",
    "x86_64-nerves-linux-musl": "x86_64-unknown-linux-musl"
  }

Then we need to check if this library is being compiled to a Nerves build. Luckily, Nerves exports a number of environment variables, and here we will try to get the value of NERVES_SDK_SYSROOT. If we get a string, then this will be a Nerves build; otherwise, we don't need to do anything special.

def project do
  if is_binary(System.get_env("NERVES_SDK_SYSROOT")) do
    components = System.get_env("CC")
      |> tap(&System.put_env("RUSTFLAGS", "-C linker=#{&1}"))
      |> Path.basename()
      |> String.split("-")

    target_triple =
      components
      |> Enum.slice(0, Enum.count(components) - 1)
      |> Enum.join("-")

    mapping = Map.get(@nerves_rust_target_triple_mapping, String.to_atom(target_triple))
    if is_binary(mapping) do
      System.put_env("RUSTLER_TARGET", mapping)
    end
  end
  
  # ... no more changes below ...
  [
      app: ... ,
      version: ...,
      ...
  ]
end

In the code above, we put a new environment variable RUSTFLAGS and set its value to -C linker=${CC}. The value of ${CC} is also automatically set by Nerves. So, here we explicitly tell rustc which linker to use as we don't want to use the linker on the host machine. ${CC} in this example is

${TOOLCHAIN_ROOT}/${NERVES_TARGET_TRIPLE}/${NERVES_TARGET_TRIPLE}-gcc

The next step of the code in project/0 is to extract and map the ${NERVES_TARGET_TRIPLE} to corresponding target in rustc. The mapped value will be set to a new environment variable RUSTLER_TARGET.

Lastly, add the target option to the use clause in the correspondingly .ex file to tell rustlter which target we are going to build.

defmodule Example.Nif do
  @moduledoc false

  # defp make_target_flag(args, target) when is_binary(target)
  # so it is fine if we get `:nil`
  use Rustler, otp_app: :exmaple, crate: "example_nif", target: System.get_env("RUSTLER_TARGET")

  def your_nif_function, do: :erlang.nif_error(:not_loaded)
end

Here you go! No more pre-compiled NIF files are needed for different Nerves build (once PR #423 is approved and merged)! An example project can be found here and also its CI build logs for almost all Nerves targets.