Month: August 2022

Precomplation support in elixir_make

There is a growing number of Elixir libraries that come with functions implemented in foreign languages like C/C++, Rust, Zig and so on for different reasons. Compiling these foreign codes isn't a big issue on a powerful machine, but for less powerful devices (limited by size, thermal, power consumption, etc.) such as a Raspberry Pi, it can take a very long time.

Also, for livebook (.livemd file) that uses Mix.install to install dependent NIF libraries, such as:

Mix.install([
  {:nif_lib_a, "~> 0.1"}
])

the nif_lib_a will be compiled and cached based on the whole deps configuration. That means if we later would like to add another library, even the newly added one is implemented in pure elixir, nif_lib_a will have to be recompiled unless nif_lib_a explicitly uses a global location to cache itself.

Mix.install([
  # nif_lib_a will be compiled again
  # even if the newly added library is
  # implemented in pure elixir
  {:nif_lib_a, "~> 0.1"},
  {:pure_elixir_lib_b, "~> 0.1"}
])

There are some other reasons that one might want to use precompiled artefacts besides the above one,

  • when the compiling toolchain is not available (e.g, running livebook on some embedded devices, or running in the nerves environment, where a C compiler is not shipped)
  • a working C/C++, Rust, or Zig compiler will not be a strict requirement.
  • to save the compilation time.

Although it's possible to completely leave the task of reusing compiled artefacts to each NIF library, there should be a way to make it at least slightly easier.

Therefore I'm exploring adding a behaviour to elixir_make to support using precompiled artefacts. And here is the Mix.Tasks.ElixirMake.Precompile behaviour (elixir-lang/elixir_make#55).

It's a behaviour because this would allow elixir_make using different precompilers for different NIF libraries to suit the needs. Also, this allows the NIF library developers to change their preferred precompiler module. And lastly, as you might've guessed, anyone can write their own precompiler module if there is no existing precompiler that quite fits their compilation pipelines.

If you'd like to write a precompiler module yourself, there are 7 required and 2 optional callbacks. It's recommended to implement them in the following order:

  • all_supported_targets/0 and current_target/0.
  @typedoc """
  Target triplets
  """
  @type target :: String.t()

  @doc """
  This callback should return a list of triplets ("arch-os-abi") for all supported targets.
  """
  @callback all_supported_targets() :: [target]

  @doc """
  This callback should return the target triplet for current node.
  """
  @callback current_target() :: {:ok, target} | {:error, String.t()}

As their name suggests, the precompiler should return the identifier of all supported targets and the current target. Usually, the identifier is the arch-os-abi triplet, but it is not a strict requirement because these identifiers are only used within the same precompiler module.

Note that it is possible for the precompiler module to pick up other environment variables like TARGET_ARCH and use them to override the value of the current target.

  • build_native/1.
  @doc """
  This callback will be invoked when the user executes the `mix compile`
  (or `mix compile.elixir_make`) command.
  The precompiler should then compile the NIF library "natively". Note that
  it is possible for the precompiler module to pick up other environment variables
  like `TARGET_ARCH=aarch64` and adjust compile arguments correspondingly.
  """
  @callback build_native(OptionParser.argv()) :: :ok | {:ok, []} | no_return

This callback corresponds to the mix compile command. The precompiler should then compile the NIF library for the current target.

After implementing this one, you can try to compile the NIF library natively with the mix compile command.

  • precompile/2.
  @typedoc """
  A map that contains detailed info of a precompiled artefact.
  - `:path`, path to the archived build artefact.
  - `:checksum_algo`, name of the checksum algorithm.
  - `:checksum`, the checksum of the archived build artefact using `:checksum_algo`.
  """
  @type precompiled_artefact_detail :: %{
    :path => String.t(),
    :checksum => String.t(),
    :checksum_algo => atom
  }

  @typedoc """
  A tuple that indicates the target and the corresponding precompiled artefact detail info.
  `{target, precompiled_artefact_detail}`.
  """
  @type precompiled_artefact :: {target, precompiled_artefact_detail}

  @doc """
  This callback should precompile the library to the given target(s).
  Returns a list of `{target, acrhived_artefacts}` if successfully compiled.
  """
  @callback precompile(OptionParser.argv(), [target]) :: {:ok, [precompiled_artefact]} | no_return

There are two arguments passed to this callback, the first one is the command line args. For example, the value of the first one will be [arg1, arg2] if one executes mix elixir_make.precompile arg1 arg2.

The second argument passed to the callback is a list of target identifiers. The precompiler should compile the NIF library for all of these targets.Note that this list of targets is always a subset of the result returned by all_supported_targets/0 but not necessarily the same list. This is designed to avoid introducing a foreseeable breaking change for precompiler modules if we prefer to add a mechanism to filter out some targets in the future.

After implementing this one, you can try the mix elixir_make.precompile command to compile for all targets.

  • post_precompile/1.
  @doc """
  This optional callback will be invoked when all precompilation tasks are done,
  i.e., it will only be called at the end of the `mix elixir_make.precompile`
  command.
  Post actions to run after all precompilation tasks are done. For example,
  actions can be archiving all precompiled artefacts and uploading the archive
  file to an object storage server.
  """
  @callback post_precompile(context :: term()) :: :ok

This is an optional callback where the precompiler module can do something after precompile/2. The argument passed to this callback is the precompiler module context.

  • precompiler_context/1.
  @doc """
  This optional callback is designed to store the precompiler's state or context.
  The returned value will be used in the `download_or_reuse_nif_file/1` and
  `post_precompile/1` callback.
  """
  @callback precompiler_context(OptionParser.argv()) :: term()

This is an optional callback that returns a custom variable which is (perhaps) based on the list of the command line arguments passed to it. The returned variable may contain anything it needs for the post_precompile/1 and download_or_reuse_nif_file/1.

For example, a simple HTTP-based username and password that are passed by command line arguments. mix elixir_make.precompile --auth USERNAME:PASSWD or mix elixir_make.fetch --all --user hello --pass world.

  • available_nif_urls/0 and current_target_nif_url/0.
  @doc """
  This callback will be invoked when the user executes the following commands:
  - `mix elixir_make.fetch --all`
  - `mix elixir_make.fetch --all --print`
  The precompiler module should return all available URLs to precompiled artefacts
  of the NIF library.
  """
  @callback available_nif_urls() :: [String.t()]

  @doc """
  This callback will be invoked when the user executes the following commands:
  - `mix elixir_make.fetch --only-local`
  - `mix elixir_make.fetch --only-local --print`
  The precompiler module should return the URL to a precompiled artefact of
  the NIF library for current target (the "native" host).
  """
  @callback current_target_nif_url() :: String.t()

The first one should return a list of URLs to the precompiled artefacts of all available targets, and the second one should return a single URL for the current target.

  • download_or_reuse_nif_file/1.
  @doc """
  This callback will be invoked when the NIF library is trying to load functions
  from its shared library.
  The precompiler should download or reuse nif file for current target.
  ## Paramters
    - `context`: Precompiler context returned by the `precompiler_context` callback.
  """
  @callback download_or_reuse_nif_file(context :: term()) :: :ok | {:error, String.t()} | no_return

The precompiler module should either download the precompiler artefacts or reuse local caches for the current target. The argument passed to this callback is the precompiler context.

Below is the Mix.Tasks.ElixirMake.Precompile behaviour (as of 04 Aug 2022). And here is a complete demo precompiler, cocoa-xu/cc_precompiler and an example project that uses this demo precompiler, cocoa-xu/cc_precompiler_example. 

defmodule Mix.Tasks.ElixirMake.Precompile do
  use Mix.Task

  @typedoc """
  Target triplets
  """
  @type target :: String.t()

  @doc """
  This callback should return a list of triplets ("arch-os-abi") for all supported targets.
  """
  @callback all_supported_targets() :: [target]

  @doc """
  This callback should return the target triplet for current node.
  """
  @callback current_target() :: {:ok, target} | {:error, String.t()}

  @doc """
  This callback will be invoked when the user executes the `mix compile`
  (or `mix compile.elixir_make`) command.
  The precompiler should then compile the NIF library "natively". Note that
  it is possible for the precompiler module to pick up other environment variables
  like `TARGET_ARCH=aarch64` and adjust compile arguments correspondingly.
  """
  @callback build_native(OptionParser.argv()) :: :ok | {:ok, []} | no_return

  @typedoc """
  A map that contains detailed info of a precompiled artefact.
  - `:path`, path to the archived build artefact.
  - `:checksum_algo`, name of the checksum algorithm.
  - `:checksum`, the checksum of the archived build artefact using `:checksum_algo`.
  """
  @type precompiled_artefact_detail :: %{
    :path => String.t(),
    :checksum => String.t(),
    :checksum_algo => atom
  }

  @typedoc """
  A tuple that indicates the target and the corresponding precompiled artefact detail info.
  `{target, precompiled_artefact_detail}`.
  """
  @type precompiled_artefact :: {target, precompiled_artefact_detail}

  @doc """
  This callback should precompile the library to the given target(s).
  Returns a list of `{target, acrhived_artefacts}` if successfully compiled.
  """
  @callback precompile(OptionParser.argv(), [target]) :: {:ok, [precompiled_artefact]} | no_return

  @doc """
  This callback will be invoked when the NIF library is trying to load functions
  from its shared library.
  The precompiler should download or reuse nif file for current target.
  ## Paramters
    - `context`: Precompiler context returned by the `precompiler_context` callback.
  """
  @callback download_or_reuse_nif_file(context :: term()) :: :ok | {:error, String.t()} | no_return

  @doc """
  This callback will be invoked when the user executes the following commands:
  - `mix elixir_make.fetch --all`
  - `mix elixir_make.fetch --all --print`
  The precompiler module should return all available URLs to precompiled artefacts
  of the NIF library.
  """
  @callback available_nif_urls() :: [String.t()]

  @doc """
  This callback will be invoked when the user executes the following commands:
  - `mix elixir_make.fetch --only-local`
  - `mix elixir_make.fetch --only-local --print`
  The precompiler module should return the URL to a precompiled artefact of
  the NIF library for current target (the "native" host).
  """
  @callback current_target_nif_url() :: String.t()

  @doc """
  This optional callback is designed to store the precompiler's state or context.
  The returned value will be used in the `download_or_reuse_nif_file/1` and
  `post_precompile/1` callback.
  """
  @callback precompiler_context(OptionParser.argv()) :: term()

  @doc """
  This optional callback will be invoked when all precompilation tasks are done,
  i.e., it will only be called at the end of the `mix elixir_make.precompile`
  command.
  Post actions to run after all precompilation tasks are done. For example,
  actions can be archiving all precompiled artefacts and uploading the archive
  file to an object storage server.
  """
  @callback post_precompile(context :: term()) :: :ok

  @optional_callbacks precompiler_context: 1, post_precompile: 1
end