Conditional compilation with "if" and "use" in Elixir

# TL; DR

If an optional module requires you to use use Module in your code, you will need to check the availability of that module before defmodule and use Module:

# The condition here does have to be Code.ensure_loaded?(OptionalModule)
# It can be Application.compile_env(:my_app, :enable_optional_feature)
# or any other conditions that can be evaluted at compile-time
if Code.ensure_loaded?(OptionalModule) do
  defmodule MyLibrary do
    use OptionalModule, option: value
    def my_function do
      OptionalModule.function()
    end
  end
else
  defmodule MyLibrary do
    def my_function do
      raise RuntimeError, """
      `OptionalModule` does not exist. To use this feature,
      please add `:optional_module` to the dependency list.
      """
    end
  end
end

and following code won't work, although it looks simpler and is seemingly correct.

defmodule MyLibrary do
  if Code.ensure_loaded?(OptionalModule) do
    use OptionalModule, option: value
    def my_function do
      OptionalModule.function()
    end
  else
    def my_function do
      raise RuntimeError, """
      `OptionalModule` does not exist. To use this feature,
      please add `:optional_module` to the dependency list.
      """
    end
  end
end

# Environment for This Post

At the time of writing, I'm using Elixir version v1.14.1. OTP version and operating system are not related to the things discussed below.

This behaviour is unlikely to change in the future, however, if you are using a future version of Elixir, please always verify the correctness/validity of the code and the conclusion in the post.

# A careless mistake?

A few hours ago I received an issue report related to evision v0.1.14 on elixirforum, and the issue can be reproduced in one line:

$ iex
iex> Mix.install({:evision, "== 0.1.14"})
...
==> evision
Compiling 179 files (.ex)

== Compilation error in file lib/smartcell/ml_traindata.ex ==
** (CompileError) lib/smartcell/ml_traindata.ex:3: module Kino.JS is not loaded and could not be found. This may be happening because the module you are trying to load directly or indirectly depends on the current module
    (elixir 1.14.0) expanding macro: Kernel.use/2
    lib/smartcell/ml_traindata.ex:3: Evision.SmartCell.ML.TrainData (module)
    (elixir 1.14.0) expanding macro: Kernel.if/2
    lib/smartcell/ml_traindata.ex:2: Evision.SmartCell.ML.TrainData (module)
could not compile dependency :evision, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile evision", update it with "mix deps.update evision" or clean it with "mix deps.clean evision"

And how did I fail to catch this, or how did this bug hide under the seemingly correct code? Is this simply a careless mistake or is there something deeper inside it? Let me unroll this story about if and use for you.

# Use if at Compile-time

Using if at compile-time is pretty common to see in Elixir. It is mainly used to define some functions under some conditions, for example

defmodule A do
  if 1 < 2 do
    def print, do: IO.puts("Yes, 1 < 2")
  else
    def print, do: IO.puts("Oh...")
  end
end

A.print

And if we run it, the expected behaviour is that the program prints Yes, 1 < 2 and exits.

$ mix run --no-mix-exs exmaple.exs
Yes, 1 < 2

Also, we can verify that the elixir compiler will remove the false branch of the if-statement (if the condition can be determined at compile-time)

defmodule A do
  if 1 < 2 do
    def should_exist, do: IO.puts("ok")
  else
    def should_not_exist, do: IO.puts("what")
  end
end

A.should_exist
A.should_not_exist
$ mix run --no-mix-exs false-branch.exs
ok
** (UndefinedFunctionError) function A.should_not_exist/0 is undefined or private. Did you mean:

      * should_exist/0

    A.should_not_exist()
    (elixir 1.14.0) lib/code.ex:1245: Code.require_file/2
    (mix 1.14.0) lib/mix/tasks/run.ex:144: Mix.Tasks.Run.run/5

Lastly, we can verify that if we use the return value of Code.ensure_loaded?/1, it will do the same thing as the above examples -- we can expect only the code under the true branch will be compiled:

# mix new ensure_loaded_example
# cd ensure_loaded_example

defmodule A do
  if Code.ensure_loaded?(B) do
    def has_b?, do: true
    def should_not_exists, do: IO.puts("what")
  else
    def has_b?, do: false
    def should_exists, do: IO.puts("ok")
  end
end

Let's try it in the IEx session, iex -S mix

$ iex -S mix
iex> A.has_b?
false
iex> defmodule B do
...> end
iex> A.has_b?
false
iex> A.should_exists
ok
:ok
iex> A.should_not_exists
** (UndefinedFunctionError) function A.should_not_exists/0 is undefined or private. Did you mean:

      * should_exists/0

    (task1b 0.1.0) A.should_not_exists()
    iex:5: (file)

# Apply What We Have Learnt So Far

Now, a quick background on the module that caused the compilation error: we'd like to write a module that implements Kino.SmartCell, and we'd like this to be an optional feature.

Therefore, we should check if Kino.SmartCell is loaded, if yes, we then follow kino's SmartCell tutorial, define our own functions and get the job done; otherwise, no functions will be defined in this module.

With the above idea in mind, we have the following code:

defmodule Evision.SmartCell.ML.TrainData do
  if Code.ensure_loaded?(Kino.SmartCell) do
    use Kino.JS, assets_path: "lib/assets"
    # ...
  end
end

At first glance, this code looks good to me because

  1. Kino.SmartCell is built on top of Kino.JS. Therefore, if Kino.SmartCell is loaded, then Kino.JS must be loaded too.
  2. we use use after we have ensured that Kino.JS is loaded.

Of course, :kino is indeed listed in deps in evision's mix.exs file.

# What Went Wrong?

Well then, let's face the inevitable question -- what went wrong?

Apparently, the elixir compiler tried to evaluate the use-statement on line 3, although we expected the compiler to completely avoid evaluating this false branch. However, since the elixir compiler has to check all the code, including the code in the false branch, is syntactically correct, e.g., the following code will not pass the syntax check:

defmodule Foo do
  if false do
    Bar.(
  end
end
$ mix run --no-mix-exs foo.exs
** (SyntaxError) foo.exs:9:3: unexpected reserved word: end. The "(" at line 8 is missing terminator ")"
    (elixir 1.14.0) lib/code.ex:1245: Code.require_file/2
    (mix 1.14.0) lib/mix/tasks/run.ex:144: Mix.Tasks.Run.run/5
    (mix 1.14.0) lib/mix/tasks/run.ex:84: Mix.Tasks.Run.run/1
    (mix 1.14.0) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.0) lib/mix/cli.ex:84: Mix.CLI.run_task/2

During this process, macros will be expanded. You might have already known that use is a macro. So yes, that's where the problem emerges.

Based on the information given in the Getting Started manual on elixir-lang.org, the following code

defmodule A do
  use Kino.JS, assets: "lib/assets"
end

will be compiled into

defmodule A do
  require Kino.JS
  Kino.JS.__using__(assets: "lib/assets")
end

Since __using__ is a macro, we have to call require to bring in all the macros defined in that module.

You see, if the application :kino is optional and the user didn't list :kino in their deps, then require will surely fail because it cannot find the Kino.JS module let alone the macros defined in Kino.JS.

And require will fail even if the use-statement (or the require-statement, after the expansion) is technically inside the false branch of an if-statement, and even when the condition is absolutely false:

defmodule A do
  if false do
    use NOT_EXISTS
  end
end
$ mix run --no-mix-exs absolutely-false.exs
** (CompileError) absolutely-false.exs:3: module NOT_EXISTS is not loaded and could not be found
    (elixir 1.14.0) expanding macro: Kernel.use/1
    example.exs:3: A (module)
    (elixir 1.14.0) expanding macro: Kernel.if/2
    example.exs:2: A (module)

Any Solution to This?

The simplest way is, of course, listing :kino as a required dependency, but I found another way to solve this issue while keeping :kino as an optional dependency.

We just saw that the following code would cause a compilation error,

defmodule A do
  if false do
    use NOT_EXISTS
  end
end

but if we slightly re-arrange these three macros (yes, defmodule, if and use they are all macros), we can achieve what we want:

if false do
  defmodule A do
    use NOT_EXISTS
  end
end

As for evision, we can change the code from

defmodule Evision.SmartCell.ML.TrainData do
  if Code.ensure_loaded?(Kino.SmartCell) do
    use Kino.JS, assets_path: "lib/assets"
    # ...
  end
end

to the following

if Code.ensure_loaded?(Kino.SmartCell) do
  defmodule Evision.SmartCell.ML.TrainData do
    use Kino.JS, assets_path: "lib/assets"
    # ...
  end
end

And...problem solved! However, this post will not end here without knowing its reasons -- why this happened in the first place? why it works after re-arranging if, defmodule and use.

# The Reasons and Explanations

After digging into Elixir's source code for a few hours, I found some clues that might be related to this. Since we placed the if-statement first and it solved the problem, I was guessing that it might have something to do with the Elixir compiler and how the compiler behaves when traversing the AST (abstract syntax tree).

  1. The first clue is about macro expansion, like when macros are expanded by the compiler.
  2. The second one is the optimize_boolean/1 function in build_if/2 in kernel.ex. Maybe the if-statement here didn't get optimised, which could cause the compilation error.
  3. The last clue I suspected is allows_fast_compilation/1 for defmodule, in elixir_compiler.erl. Because the name of this function is kinda sus, and based on the name, it seems to allow the compiler to skip/defer the evaluation of defmodule.
allows_fast_compilation({defmodule, _, [_, [{do, _}]]}) ->
  true;

Then I decided to ask @josevalim about this question, and he replied kindly, and his answers solved this puzzle. I'll now connect all the dots along with his answers and explain this mystery below.

A big thanks to José!

In Elixir's source code (v1.14.1), defmodule is defined in kernel.ex, 

defmacro defmodule(alias, do_block)

use is in kernel.ex too

defmacro use(module, opts \\ [])

and, of course, as spoiled above, if is also a macro

defmacro if(condition, clauses)

And macros are always expanded before the branches (e.g., an if-statement has two branches, true and false) are evaluated. That is to say, when the compiler sees the following code

defmodule A do
  if false do
    use NOT_EXISTS
  end
end

It will expand the macros first:

defmodule A do
  if false do
    require NOT_EXISTS
    NOT_EXISTS.__using__([])
  end
end

But it will fail when evaluating require NOT_EXISTS because the module does not exist. We can see this based on the traced stack:

When the compiler sees an if-statement, it will go inside and expand macros (if any). That's why the Kernel.use/1 is on top of Kernel.if/2 in the traced stack:

$ mix run --no-mix-exs absolutely-false.exs
** (CompileError) absolutely-false.exs:3: module NOT_EXISTS is not loaded and could not be found
    (elixir 1.14.0) expanding macro: Kernel.use/1
    example.exs:3: A (module)
    (elixir 1.14.0) expanding macro: Kernel.if/2
    example.exs:2: A (module)

And then I asked why putting if before defmodule will not cause the same compilation error. And José answered:

Because there are a few things that delay macro expansion, such as defining new modules or defining functions. Because you can do this:

module = Foo
defmodule module do
  ...
end

Therefore, to define the module, you must execute the line that defines it. So only after you execute the line, the macros are expanded. defmodule A do ... end compiles to something like this:

:elixir_module.compile(A, ast_of_the_module_body)

that AST will only be expanded if the module is in fact defined.

Therefore, the following code 

defmodule A do
  if false do
    use NOT_EXISTS
  end
end

is first compiled to the following quoted code in kernel.ex

quote do
  unquote(with_alias)
  :elixir_module.compile(unquote(expanded), unquote(escaped), unquote(module_vars), __ENV__)
end

And this emitted code will be executed because the defmodule in this example is unconditionally written in the source code instead of existing in the false branch of an if-statement. When :elixir_module.compile/4 is invoked, the if-statement will be evaluated, and the use macro will be expanded during the evaluation process of if.

Again, we can verify this by looking at the stacktrace:

$ mix run --no-mix-exs absolutely-false.exs
** (CompileError) absolutely-false.exs:3: module NOT_EXISTS is not loaded and could not be found
    (elixir 1.14.0) expanding macro: Kernel.use/1
    example.exs:3: A (module)
    (elixir 1.14.0) expanding macro: Kernel.if/2
    example.exs:2: A (module)

Searching is not loaded and could not be found in Elixir's source code will bring us to format_error/1 in elixir_aliases.erl.

format_error({unloaded_module, Module}) ->
  io_lib:format("module ~ts is not loaded and could not be found", [inspect(Module)]);

Since unloaded_module is an atom, we can search it and trace back where it was produced. And it only appears in the same file in ensure_loaded/3.

ensure_loaded(Meta, Module, E) ->
  case code:ensure_loaded(Module) of
    {module, Module} ->
      ok;

    _ ->
      case wait_for_module(Module) of
        found ->
          ok;

        Wait ->
          Kind = case lists:member(Module, ?key(E, context_modules)) of
            true ->
              case ?key(E, module) of
                Module -> circular_module;
                _ -> scheduled_module
              end;
            false when Wait == deadlock ->
              deadlock_module;
            false ->
              unloaded_module
          end,

          elixir_errors:form_error(Meta, E, ?MODULE, {Kind, Module})
      end
  end.

This function will first check if the module is already loaded, if not, it will invoke wait_for_module/1 and wait for the module to be compiled if that module exists and can be compiled.

wait_for_module(Module) ->
  case erlang:get(elixir_compiler_info) of
    undefined -> not_found;
    _ -> 'Elixir.Kernel.ErrorHandler':ensure_compiled(Module, module, hard)
  end.

Yet obviously module NOT_EXISTS does not exist, and 'Elixir.Kernel.ErrorHandler':ensure_compiled/3 will return unloaded_module, which will be subsequently passed to elixir_errors:form_error/4.

form_error(Meta, #{file := File}, Module, Desc) ->
  compile_error(Meta, File, Module:format_error(Desc));
form_error(Meta, File, Module, Desc) ->
  compile_error(Meta, File, Module:format_error(Desc)).

and become the compilation error message.

** (CompileError) absolutely-false.exs:3: module NOT_EXISTS is not loaded and could not be found

And to dig deeper, we can search :ensure_loaded in the code base and the one in elixir_expand.erl invoked in function expand/3 are what we are interested in.

expand({require, Meta, [Ref, Opts]}, S, E) ->
  assert_no_match_or_guard_scope(Meta, "require", S, E),

  {ERef, SR, ER} = expand_without_aliases_report(Ref, S, E),
  {EOpts, ST, ET}  = expand_opts(Meta, require, [as, warn], no_alias_opts(Opts), SR, ER),

  if
    is_atom(ERef) ->
      elixir_aliases:ensure_loaded(Meta, ERef, ET),
      {ERef, ST, expand_require(Meta, ERef, EOpts, ET)};
    true ->
      form_error(Meta, E, ?MODULE, {expected_compile_time_module, require, Ref})
  end;

And this is exactly where the compilation error happens. 

defmodule A do
  if false do
    require NOT_EXISTS       # <= elixir_aliases:ensure_loaded(Meta, ERef, ET),
    NOT_EXISTS.__using__([])
  end
end

And we can continue to search elixir_expand:expand in the source code, and we will find the function expand_quoted/7 in elixir_dispatch.erl.

expand_quoted(Meta, Receiver, Name, Arity, Quoted, S, E) ->
  Next = elixir_module:next_counter(?key(E, module)),

  try
    ToExpand = elixir_quote:linify_with_context_counter(Meta, {Receiver, Next}, Quoted),
    elixir_expand:expand(ToExpand, S, E)
  catch
    Kind:Reason:Stacktrace ->
      MFA  = {Receiver, elixir_utils:macro_name(Name), Arity+1},
      Info = [{Receiver, Name, Arity, [{file, "expanding macro"}]}, caller(?line(Meta), E)],
      erlang:raise(Kind, Reason, prune_stacktrace(Stacktrace, MFA, Info, error))
  end.

Note that the Info on line 248 will become

    ...
    (elixir 1.14.0) expanding macro: Kernel.use/1
    ...
    (elixir 1.14.0) expanding macro: Kernel.if/2
    ...

in the stacktrace. One more step and we will find that expand_quoted/7 is called in two dispatch functions: dispatch_import/6 and dispatch_require/7 in the same file.

dispatch_require(Meta, Receiver, Name, Args, S, E, Callback) when is_atom(Receiver) ->
  Arity = length(Args),

  case elixir_rewrite:inline(Receiver, Name, Arity) of
    {AR, AN} ->
      Callback(AR, AN, Args);
    false ->
      case expand_require(Meta, Receiver, {Name, Arity}, Args, S, E) of
        {ok, Receiver, Quoted} -> expand_quoted(Meta, Receiver, Name, Arity, Quoted, S, E);
        error -> Callback(Receiver, Name, Args)
      end
  end;

dispatch_require(_Meta, Receiver, Name, Args, _S, _E, Callback) ->
  Callback(Receiver, Name, Args).

As for elixir_dispatch:dispatch_require/7, it is invoked in two places: the first one is when processing remote calls in elixir_expand:expand/3. A remote call is to invoke/call a function in another module other than the current one. For example,

defmodule A do
  def foo, do: :ok

  def bar do
    # this is a local call
    foo()
  end
end

defmodule B do
  def baz do
    # this is a remote call
    A.foo()
  end
end

The second call to elixir_dispatch:dispatch_require/7 can be found in elixir_module:expand_callback/6.

expand_callback(Line, M, F, Args, Acc, Fun) ->
  E = elixir_env:reset_vars(Acc),
  S = elixir_env:env_to_ex(E),
  Meta = [{line, Line}, {required, true}],

  {EE, _S, ET} =
    elixir_dispatch:dispatch_require(Meta, M, F, Args, S, E, fun(AM, AF, AA) ->
      Fun(AM, AF, AA),
      {ok, S, E}
    end),

  if
    is_atom(EE) ->
      ET;
    true ->
      try
        {_Value, _Binding, EF} = elixir:eval_forms(EE, [], ET),
        EF
      catch
        Kind:Reason:Stacktrace ->
          Info = {M, F, length(Args), location(Line, E)},
          erlang:raise(Kind, Reason, prune_stacktrace(Info, Stacktrace))
      end
  end.

elixir_module:expand_callback/6 is used in two places: the first one is in Protocol.derive/5, and as the module name suggested, it is related to deriving a protocol for a module.

We can go off on a tangent here and explore what will happen if we want to derive a protocol optionally. You can skip this and jump to the second call to elixir_module:expand_callback/6.

Let's say we have already defined a protocol Derivable as the following:

defprotocol Derivable do
  def ok(arg)
end

defimpl Derivable, for: Any do
  defmacro __deriving__(module, struct, options) do
    quote do
      defimpl Derivable, for: unquote(module) do
        def ok(arg) do
          {:ok, arg, unquote(Macro.escape(struct)), unquote(options)}
        end
      end
    end
  end

  def ok(arg) do
    {:ok, arg}
  end
end

Then there are three ways to derive a protocol for a module, and we can start with the simplest one:

  • Deriving a protocol using the @derive tag.
defmodule A do
  @derive {Derivable, option: :some_value}
  defstruct a: 0, b: 0
end
  • The second way is using the defimpl/3 macro in kernel.ex.
defmodule A do
  defimpl Derivable do
    def ok(_arg), do: :my_args
  end
end
defmodule A do
  defstruct a: 0, b: 0

  require Protocol
  Protocol.derive(Derivable, A, option: :some_value)
end

What will happen if we'd like to derive a protocol from an optional module?

defmodule A do
  if Code.ensure_loaded?(NOT_EXISTS) do
    @derive {NOT_EXISTS, option: :some_value}
  end
  defstruct a: 0, b: 0
end

That's the first one, and the second one is shown below

defmodule A do
  if Code.ensure_loaded?(NOT_EXISTS) do
    defimpl NOT_EXISTS, for: A do
    end
  end
  defstruct a: 0, b: 0
end

And the last one

defmodule A do
  defstruct a: 0, b: 0
  
  if Code.ensure_loaded?(NOT_EXISTS) do
    require Protocol
    Protocol.derive(NOT_EXISTS, A, option: :some_value)
  end
end

All three examples that optionally derive a protocol from an optional module will compile without any issues and behave as expected. Of course, we'd like to ask the question -- why can we use if inside the defmodule to optionally derive a protocol?

Let me explain the reasons for the first way first. The @derive tag is a module attribute, and @derive tags in the same module will be collected accumulatively in a bag -- simply put, they will be stored in a list. Then the defstruct/1 macro will retrieve them from Kernel.Utils.defstruct/3 and rewrite them with Protocol.__derive__/3. (This also explains why all @derive tags must be set before defstruct/1)

defmacro defstruct(fields) do
  quote bind_quoted: [fields: fields, bootstrapped?: bootstrapped?(Enum)] do
    {struct, derive, kv, body} = Kernel.Utils.defstruct(__MODULE__, fields, bootstrapped?)

    case derive do
      [] -> :ok
      _ -> Protocol.__derive__(derive, __MODULE__, __ENV__)
    end

    def __struct__(), do: @__struct__
    def __struct__(unquote(kv)), do: unquote(body)

    Kernel.Utils.announce_struct(__MODULE__)
    struct
  end
end

Since Protocol.__derive__/3 is a function instead of a macro, it will not be evaluated/expanded when processing the branches of an if-statement. Therefore, the following code will compile and run as expected.

defmodule A do
  if Code.ensure_loaded?(NOT_EXISTS) do
    @derive {NOT_EXISTS, option: :some_value}
  end
  defstruct a: 0, b: 0
end

As for the second way that uses the defimpl/3 macro in kernel.ex, although defimpl/3 is a macro, it simply rewrites the code to call the Protocol.__impl__/4 function.

defmacro defimpl(name, opts, do_block \\ []) do
  Protocol.__impl__(name, opts, do_block, __CALLER__)
end

So, it's similar to the first one, the call to the function Protocol.__impl__/4 will not be evaluated when processing the branches of an if-statement.

The same reason goes for the last one: Protocol.derive/3 is a function, so it will not be evaluated when processing the branches of an if-statement.

Let's get back on track. The second call to elixir_module:expand_callback/6 can be found in eval_callbacks/5 in elixir_module.erl.

eval_callbacks(Line, DataBag, Name, Args, E) ->
  Callbacks = bag_lookup_element(DataBag, {accumulate, Name}, 2),
  lists:foldl(fun({M, F}, Acc) ->
    expand_callback(Line, M, F, Args, Acc, fun(AM, AF, AA) -> apply(AM, AF, AA) end)
  end, E, Callbacks).

eval_callbacks/5 is called in the same file in two different functions (actually one function, depending on how you see this). The first line it appears in the file is inside the compile/5 function (line 161). The second time it appears is in the eval_form/6 function (line 378).

However, eval_form/6 is only called once from the compile/5 function. The obvious difference is that the third argument passed by eval_form/6 is before_compile whereas compile/5 passes after_compile. So I guess you could say that eval_callbacks/5 is called in one function, compile/5.

compile(Line, Module, Block, Vars, E) ->
  File = ?key(E, file),
  check_module_availability(Line, File, Module),
  ModuleAsCharlist = validate_module_name(Line, File, Module),

  CompilerModules = compiler_modules(),
  {Tables, Ref} = build(Line, File, Module),
  {DataSet, DataBag} = Tables,

  try
    put_compiler_modules([Module | CompilerModules]),
    {Result, NE} = eval_form(Line, Module, DataBag, Block, Vars, E),
    CheckerInfo = checker_info(),
    ...

Based on the value of the third argument, and what we have is a compilation error, we can confirm that the compilation error is thrown when calling eval_form/6 with the third argument as before_compile.

And we are very close to connecting all the dots: eval_form/6 is called from elixir_module:compile/5, and elixir_module:compile/5 is called in elixir_module:compile/4, which is exactly the code that defmodule rewrites to!

So, if eval_callbacks/5 in eval_form/6 is successfully executed, and there are no other errors, we can expect the optimize_boolean/1 function in build_if/2

Connecting the Dots

Let's first reproduce the whole process with the code that will cause a compilation error.

defmodule A do
  if false do
    use NOT_EXISTS
  end
end

And we start from the shell command mix run --no-mix-exs absolutely-false.exs. The entry point for mix run [argv...] is Mix.Tasks.Run.run/1, and it will parse command line arguments and call run/5. 

def run(args) do
  {opts, head} =
    OptionParser.parse_head!(
      args,
      aliases: [r: :require, p: :parallel, e: :eval, c: :config],
      strict: [
        parallel: :boolean,
        require: :keep,
        eval: :keep,
        config: :keep,
        mix_exs: :boolean,
        halt: :boolean,
        compile: :boolean,
        deps_check: :boolean,
        start: :boolean,
        archives_check: :boolean,
        elixir_version_check: :boolean,
        parallel_require: :keep,
        preload_modules: :boolean
      ]
    )

  run(args, opts, head, &Code.eval_string/1, &Code.require_file/1)
  unless Keyword.get(opts, :halt, true), do: System.no_halt(true)
  Mix.Task.reenable("run")
  :ok
end

In this example, {opts, head} will be

{[mix_exs: false], ["absolutely-false.exs"]}

In run/5, we can ignore other checks and focus on the call to the callback function file_evaulator.(file). And since file_evaulator is just &Code.require_file/1, we can jump to the function require_file/1 in the code.ex file.

def require_file(file, relative_to \\ nil) when is_binary(file) do
  {charlist, file} = find_file!(file, relative_to)

  case :elixir_code_server.call({:acquire, file}) do
    :required ->
      nil

    :proceed ->
      loaded =
        Module.ParallelChecker.verify(fn ->
          :elixir_compiler.string(charlist, file, fn _, _ -> :ok end)
        end)

      :elixir_code_server.cast({:required, file})
      loaded
  end
end

find_file!/2 expands the relative path to the absolute path of the file and reads the whole file into a char list. Then :elixir_code_server.call/1 will check if the file is already required (compiled), if yes, we will do nothing; if not compiled yet, :elixir_compiler.string/3 will be called to compile the code.

string(Contents, File, Callback) ->
  Forms = elixir:'string_to_quoted!'(Contents, 1, 1, File, elixir_config:get(parser_options)),
  quoted(Forms, File, Callback).

elixir:'string_to_quoted!'/5 will pass the file content to the tokenizer and convert tokens to their quoted form.

'string_to_quoted!'(String, StartLine, StartColumn, File, Opts) ->
  case string_to_tokens(String, StartLine, StartColumn, File, Opts) of
    {ok, Tokens} ->
      case tokens_to_quoted(Tokens, File, Opts) of
        {ok, Forms} ->
          Forms;
        {error, {Meta, Error, Token}} ->
          elixir_errors:parse_error(Meta, File, Error, Token, {String, StartLine, StartColumn})
      end;
    {error, {Meta, Error, Token}} ->
      elixir_errors:parse_error(Meta, File, Error, Token, {String, StartLine, StartColumn})
  end.

The following AST will be emitted for this code:

{:defmodule, [line: 1],
 [
   {:__aliases__, [line: 1], [:A]},
   [
     do: {:if, [line: 2],
      [
        false,
        [do: {:use, [line: 3], [{:__aliases__, [line: 3], [:NOT_EXISTS]}]}]
      ]}
   ]
 ]}

Next, elixir_compiler:quoted/3 will process the quoted form -- evaluate or compile the code with the local lexical environment in eval_or_compile/3.

quoted(Forms, File, Callback) ->
  Previous = get(elixir_module_binaries),

  try
    put(elixir_module_binaries, []),
    Env = (elixir_env:new())#{line := 1, file := File, tracers := elixir_config:get(tracers)},

    elixir_lexical:run(
      Env,
      fun (LexicalEnv) -> eval_or_compile(Forms, [], LexicalEnv) end,
      fun (#{lexical_tracker := Pid}) -> Callback(File, Pid) end
    ),

    lists:reverse(get(elixir_module_binaries))
  after
    put(elixir_module_binaries, Previous)
  end.

The implementation of eval_or_compile/3 is:

eval_or_compile(Forms, Args, E) ->
  case (?key(E, module) == nil) andalso allows_fast_compilation(Forms) andalso
        (not elixir_config:is_bootstrap()) of
    true  -> fast_compile(Forms, E);
    false -> compile(Forms, Args, E)
  end.

allows_fast_compilations/1 will check if this file always defines a module, if yes, we can skip some steps in compile/3 and always define the module.

allows_fast_compilation({'__block__', _, Exprs}) ->
  lists:all(fun allows_fast_compilation/1, Exprs);
allows_fast_compilation({defmodule, _, [_, [{do, _}]]}) ->
  true;
allows_fast_compilation(_) ->
  false.

And the AST shown above indeed matches the one in the middle, therefore, we can do fast_compile/2.

fast_compile({defmodule, Meta, [Mod, [{do, TailBlock}]]}, NoLineE) ->
  E = NoLineE#{line := ?line(Meta)},

  Block = {'__block__', Meta, [
    {'=', Meta, [{result, Meta, ?MODULE}, TailBlock]},
    {{'.', Meta, [elixir_utils, noop]}, Meta, []},
    {result, Meta, ?MODULE}
  ]},

  Expanded = case Mod of
    {'__aliases__', _, _} ->
      case elixir_aliases:expand_or_concat(Mod, E) of
        Receiver when is_atom(Receiver) -> Receiver;
        _ -> 'Elixir.Macro':expand(Mod, E)
      end;

    _ ->
      'Elixir.Macro':expand(Mod, E)
  end,

  ContextModules = [Expanded | ?key(E, context_modules)],
  elixir_module:compile(Expanded, Block, [], E#{context_modules := ContextModules}).

After the pattern matching, we get

Meta = [line: 1]
Mod = {:__aliases__, [line: 1], [:A]}
TailBlock = 
  {:if, [line: 2],
    [
      false,
      [do: {:use, [line: 3], [{:__aliases__, [line: 3], [:NOT_EXISTS]}]}]
    ]}

Now we can jump right into elixir_module:compile/4 which will call elixir_module:compile/5. And expect the compilation error right after calling elixir_module:eval_form/6 in elixir_module:compile/5. And you know the rest of the story.

What about the code that can achieve our goal?

if false do
  defmodule A do
    use NOT_EXISTS
  end
end

The AST of the above code is

{:if, [line: 1],
 [
   false,
   [
     do: {:defmodule, [line: 2],
      [
        {:__aliases__, [line: 2], [:A]},
        [do: {:use, [line: 3], [{:__aliases__, [line: 3], [:NOT_EXISTS]}]}]
      ]}
   ]
 ]}

And now the difference is that we cannot do fast_compile/2 because the file may not define a module. Therefore, now we will take the compile/3 route in eval_or_compile/3.

compile(Quoted, ArgsList, E) ->
  {Expanded, SE, EE} = elixir_expand:expand(Quoted, elixir_env:env_to_ex(E), E),
  elixir_env:check_unused_vars(SE, EE),

  {Module, Fun, Purgeable} =
    elixir_erl_compiler:spawn(fun() -> spawned_compile(Expanded, E) end),

  Args = list_to_tuple(ArgsList),
  {dispatch(Module, Fun, Args, Purgeable), EE}.

And the variable Expanded will be something like this

{:case, [line: 1, optimize_boolean: true],
 [
   false,
   [
     do: [
       {:->, [generated: true, line: 1], [[false], nil]},
       {:->, [generated: true, line: 1],
        [
          [true],
          {:__block__, [line: 2],
           [
             A,
             {{:., [line: 2], [:elixir_module, :compile]}, [line: 2],
              [
                ...
              ]}
           ]}
        ]}
     ]
   ]
 ]}

As we can see from this intermediate output, optimize_boolean/1 has already been called and gives us the tuple {:case, [line: 1, optimize_boolean: true], [false, [do: ...]]}.

In spawned_compile/2, we will translate quoted elixir expressions and variables to the forms that erlang can compile them into .beam binary.

spawned_compile(ExExprs, #{line := Line, file := File} = E) ->
  {Vars, S} = elixir_erl_var:from_env(E),
  {ErlExprs, _} = elixir_erl_pass:translate(ExExprs, erl_anno:new(Line), S),

  Module = retrieve_compiler_module(),
  Fun  = code_fun(?key(E, module)),
  Forms = code_mod(Fun, ErlExprs, Line, File, Module, Vars),

  {Module, Binary} = elixir_erl_compiler:noenv_forms(Forms, File, [nowarn_nomatch, no_bool_opt, no_ssa_opt]),
  code:load_binary(Module, "", Binary),
  {Module, Fun, is_purgeable(Module, Binary)}.

In function is_purgeable/2 we test if the beam binary has any labeled locals, 

is_purgeable(Module, Binary) ->
  beam_lib:chunks(Binary, [labeled_locals]) == {ok, {Module, [{labeled_locals, []}]}}.

If not, we can evaluate the code and return the evaluated result as the compiled output (in function dispatch/4, and dispatch/4 was called in the compile/3 route).

dispatch(Module, Fun, Args, Purgeable) ->
  Res = Module:Fun(Args),
  code:delete(Module),
  Purgeable andalso code:purge(Module),
  return_compiler_module(Module, Purgeable),
  Res.

In this example, Res will be nil because the condition used in the if-statement is false (and the false branch can be purged); meanwhile, as there is no true branch, the result of the true branch will be evaluated to the default value, nil.

Therefore, if we were using Code.ensure_loaded?/1 as the condition, it would also be evaluated to either true or false at compile-time. If it is false, then everything in the false branch will be purged, so we won't encounter any compilation errors for the following code.

if Code.ensure_loaded?(NOT_EXISTS) do
  defmodule A do
    use NOT_EXISTS
  end
end

will be evaluated to 

if false do
  defmodule A do
    use NOT_EXISTS
  end
else
  nil
end

# actually, it will be evaluated to something like this
case false do
  false -> 
    defmodule A do
      use NOT_EXISTS
    end
  true ->
    nil
end

and the false branch will be purged, which leaves us only nil

nil

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.