Macro Madness: How to use `use` well

By: Luke Imhoff
Two researchers in heavy winter gear holding flash lights and wearing headlamps overlook a collection of strange circular ruins on the cliff face and valley floor below the ledge the researchers stand on.

In Elixir, macros are used to define things that would be keywords in other languages: defmodule, def, defp, defmacro, and defmacrop are all macros defined by the standard library in Kernel. In ExUnit, assert is able to both run the code passed to it to see if the test is passing, but also print that code when it fails, so that you don’t need a custom DSL to show what was being tested. In GenServer, use GenServer defines default implementations for all the required callbacks.

If you want a head-trip, look at the implementation of defmacro, which is defined using defmacro:

defmacro defmacro(call, expr \\ nil) do
  define(:defmacro, call, expr, __CALLER__)
end

Kernel.defmacro/2

Don’t worry, like all languages defined in themselves, defmacro is defined using a “bootstrap” library that’s written in the underlying language, in Elixir’s case :elixir_bootstrap defines minimal versions of @, defmodule, def, defp, defmacro, defmacrop in Erlang: just enough for Kernel to be parsed once and then it defines the full version. This way, you don’t need the last version of Elixir to build the next version, just Erlang.

import Kernel, except: [@: 1, defmodule: 2, def: 1, def: 2, defp: 2,
                        defmacro: 1, defmacro: 2, defmacrop: 2]
import :elixir_bootstrap

Kernel

Macros allow us to generate code dynamically at compile time. One of the reasons they were added to Elixir was to reduce the amount of boiler plate needed to be written for behaviours, such as :gen_server. In Erlang, this boiler plate was manually added to each file using Emacs templates.

Before the introduction of -optional_callbacks attribute in Erlang 20, there was no way to add new callbacks without having everyone update their code to add their own copy of the default implementation.

GenServer has 6 callbacks you need to implement. Every GenServer you use would need to have the correct signature and return values for all those callbacks.

So, to implement the bare minimum, we can get away with one-liners in most cases, but we need to remember the shape of each of the returns even if we don’t care about code_change/3 for hot-code upgrades. Additionally, the one-liners with raise won’t type check with dialyzer: it will warn about non-local return, which is just dialyzer’s way of saying you’re raising an exception or throwing. The real code in GenServer is doing more to make dialyzer happy and to give you more helpful error messages that are easier to debug.

def init(args), do: {:ok, args}

def handle_call(msg, _from, state), do: raise "Not implemented"

def handle_info(msg, state) do
  :error_logger.error_msg(
    '~p ~p received unexpected message in handle_info/2: ~p~n',
   [__MODULE__, self(), msg]
  )
  {:noreply, state}
end

def handle_cast(msg, state), do: raise "Not implemented"

def terminate(_reason, _state), do: :ok

def code_change(_old, state, _extra), do: {:ok, state}

But, if you read the docs for GenServer and know that you don’t need to implement all the callbacks, you can put use GenServer in your callback module and all those default implementation will be defined for you. So, you go from having to hap-hazardly copy default implementations to each callback module to a single line.

Just like defmodule and the various def* for call definitions, use is not a keyword in Elixir, it is a macro in Kernel, so think of use as a convention, not a keyword.

use is not magic. It’s very short piece of code that is only complex to give some convenience:

  1. It automatically does require, as __using__ is a macro and macros can’t be used without an explicit require first
  2. It uses Enum.map, so you can pass multiple aliases (use Namespace.{Child1, Child2})
  3. It raises an ArgumentError if you called it wrong.
defmacro use(module, opts \\ []) do
  calls = Enum.map(expand_aliases(module, __CALLER__), fn
    expanded when is_atom(expanded) ->
      quote do
        require unquote(expanded)
        unquote(expanded).__using__(unquote(opts))
      end
    _otherwise ->
      raise ArgumentError, "invalid arguments for use, expected a compile time atom or alias, got: #{Macro.to_string(module)}"
  end)
  quote(do: (unquote_splicing calls))
end

Kernel.use/2

If use just calls the __using__ macro, what is the __using__ macro supposed to do? The only requirement is that it behaves like any other macro: it returns quoted code. The rest is up to the conventions and best practices in the docs for Kernel.use.

Example

Let’s look at an example of using __using__ and the misteps you can make along the way and how to fix them.

An Old One

While working at Miskatonic University, William Dyer started a compendium of various species the university had encountered. The university’s not mad enough to try to bring them to Earth, so we use a Client library to establish communication with grad students working in the field.

defmodule Miskatonic.OldOnes do
  def get(id) do
    with {:ok, client_pid} <- client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end

  defp client_start_link do
    Miskatonic.Clients.Portal.start_link(entrance: "witch-house")
  end
end

Miskatonic.OldOnes@william-dyer

The heads of multiple Great Old Ones merge organically with Cthulhu's head at the base

While researching the Old Ones, Miskatonic grad students found some of their records referring to greater species that the Old Ones were studying. Because naming is hard, Miskatonic has started to call them Great Old Ones.

defmodule Miskatonic.GreatOldOnes do
  def get(id) do
    with {:ok, client_pid} <- client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end

  defp client_start_link do
    Miskatonic.Clients.Boat.start_link(
      latitude: -47.15,
      longitude: -126.72
    )
  end
end

Miskatonic.GreatOldOnes@gustaf-johansen

So, we have two modules, that both have a get function for getting the research on a resource, but how we can communicate with the grad students in the fields differ. We want to make communicating with new and exciting things that want to drive us mad easier because we keep losing grad students, so we need to refactor our two modules and extract the common pieces. Here’s the general shape. There’s a get/1 function that takes an id and then internally there’s client_start_link/0 function that hides the different ways we communicate with the realms of the different species.

defmodule Miskatonic.Species do
  def get(id) do
    with {:ok, client_pid} <- client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end

  defp client_start_link do
    ??
  end
end

Using use

Using the use convention, we can move get/1 definition into a quote block in the __using__ macro for a new, general Miskatonic.Species module. We can move get/1 into it, but we can’t move client_start_link in it.

defmodule Miskatonic.Species do
  defmacro __using__([]) do
    quote do
      def get(id) do
        with {:ok, client_pid} <- client_start_link() do
          Miskatonic.Client.show(client_pid, id)
        end
      end
    end
  end
end

Miskatonic.Species@bob-howard

Now we can use Miskatonic.Species allow us to get rid of the duplicate get/1 code in each module, but we still need the client_start_link since it differs in each.

defmodule Miskatonic.OldOnes do
  use Miskatonic.Species

  defp client_start_link do
    Miskatonic.Clients.Portal.start_link(entrance: "witch-house")
  end
end

Miskatonic.OldOnes@bob-howard

defmodule Miskatonic.GreatOldOnes do
  use Miskatonic.Species

  defp client_start_link do
    Miskatonic.Clients.Boat.start_link(latitude: -47.15,
                                       longitude: -126.72)
  end
end

Miskatonic.GreatOneOne@bob-howard

Bob Howard in a tactical turtleneck holding a glow hand-held device

Bob Howard gets pulled off the project and sent to The Laundry, so a new grad student, Carly Rae Jepsen needs contact with the Yithians, who Old Ones fought.

Great Race of Yith

Seeing how useful use Miskatonic.Species was in the other modules, the Carly Rae Jepsen tries the same, but she get a cryptic error message that client_start_link/0 is undefined.

defmodule Miskatonic.Yithians do
  use Miskatonic.Species
end

Miskatonic.Yithians@carly-rae-jepsen-compilation-error

== Compilation error in file lib/miskatonic/yithians.ex ==
** (CompileError) lib/miskatonic/yithians.ex:2: undefined function client_start_link/0
    (stdlib) lists.erl:1338: :lists.foreach/2
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6

mix compile

Carly Rae tracks down that Miskatonic.Species depends on client_start_link/0 being defined, but Miskatonic.Species isn’t currently making the best use of the compiler to tell developers that. Using @callback, to declare that client_start_link/0 is required by @behaviour Miskatonic.Species that Carly Rae adds to the quote block.

defmodule Miskatonic.Species do
  @callback client_start_link() ::
              {:ok, pid} | {:error, reason :: term}

  defmacro __using__([]) do
    quote do
      @behaviour Miskatonic.Species

      def get(id) do
        with {:ok, client_pid} <- client_start_link() do
          Miskatonic.Client.show(client_pid, id)
        end
      end
    end
  end
end

Miskatonic.Species@carly-rae-jepsen-client-start-link-callback

So, great, Carly Rae gets a compiler warning now, that’s more specific about why Carly Rae needs client_start_link in Miskatonic.Yithians, but it looks like @callback implementations need to be public, so change all the defp client_start_link to def client_start_link

warning: undefined behaviour function client_start_link/0  (for behaviour Miskatonic.Species)
  lib/miskatonic/great_old_ones.ex:1

warning: undefined behaviour function client_start_link/0  (for behaviour Miskatonic.Species)
  lib/miskatonic/yithians.ex:1

warning: undefined behaviour function client_start_link/0  (for behaviour Miskatonic.Species)
  lib/miskatonic/old_ones.ex:1

mix compile

With the switch to public client_start_link/0, we can learn about the Old Ones, Great Old Ones, and Yithians, but the code could be better. Although we’re not writing the def get in every file, it’s being stored in each, which we can see if we ask for the debug info. For one function, this isn’t a big deal, but if we add more and more functions, this is unnecessary bloat, we know it’s exactly the same code. Code loading still takes time with the BEAM even if it’s faster than languages that need to be interpreted from source first.

iex> {:ok, {module, [debug_info: {_version, backend, data}]}} = :beam_lib.chunks('_build/dev/lib/miskatonic/ebin/Elixir.Miskatonic.Yithians.beam',[:debug_info])
iex> {:ok, debug_info} = backend.debug_info(:elixir_v1, module, data, [])
iex> {:ok, %{definitions: definitions}} = backend.debug_info(:elixir_v1, module, data, [])
iex> List.keyfind(definitions, {:get, 1}, 0)
{:get, 1}, :def, [line: 2, generated: true],
 [{[line: 2, generated: true],
   [{:id, [counter: -576460752303423100, line: 2], Miskatonic.Species}], [],
   {:with, [line: 2],
    [{:<-, [line: 2],
      [{:ok,
        {:client_pid, [counter: -576460752303423100, line: 2],
         Miskatonic.Species}}, {:client_start_link, [line: 2], []}]},
     [do: {{:., [line: 2], [Miskatonic.Client, :show]}, [line: 2],
       [{:client_pid, [counter: -576460752303423100, line: 2],
         Miskatonic.Species},
        {:id, [counter: -576460752303423100, line: 2],
         Miskatonic.Species}]}]]}}]}

The general approach you want to take when making functions in your __using__ quote block to be as short as possible. To do this, I recommend immediately calling a normal function in the outer module that takes __MODULE__ as an argument.

The reason I recommended always passing in the __MODULE__ is illustrated well here, module is needed, so that client_start_link/0 can be called in get/2 because it’s outside the quote block and won’t be in the module that calls use Miskatonic.Species anymore.

defmodule Miskatonic.Species do
  @callback client_start_link() ::
              {:ok, pid} | {:error, reason :: term}

  defmacro __using__([]) do
    quote do
      @behaviour Miskatonic.Species

      def get(id), do: Miskatonic.Species.get(__MODULE__, id)
    end
  end

  def get(module, id) do
    with {:ok, client_pid} <- module.client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end
end

Miskatonic.Species@get-module

Carly Rae Jepsen is doing such a good job on the code that the university doesn’t want to risk her going mad in the field, so Miskatonic University has decided to fund another graduate position on the team. Nathaniel Wingate Peaslee joins the team and discovers that the Yithian psychic link isn’t limited to just swamping location, but can be used to swap in time. This means to study more of Yithians, the Miskatonic.Yithians module should try mind transferring to a Yithian in a different time, if getting info on a Yithian fails.

defmodule Miskatonic.Yithians do
  use Miskatonic.Species

  def client_start_link(keywords \\ [yithian: "Librarian"]) do
    Miskatonic.Clients.Psychic.start_link(keywords)
  end

  def get(id) do
    case Miskatonic.Species.get(__MODULE__, id) do
      {:error, :not_found} ->
        with {:ok, pid} <- client_start_link(yithian: "Coleopterous") do
          Miskatonic.Client.show(pid, id)
        end
      found ->
        found
    end
  end
end

Miskatonic.Yithians@clause-cannot-match

Ah, but Nathaniel seems unable to override get/1 that the use Miskatonic.Species is inserting. Line 2 is the line where use Miskatonic.Species is called while line 8 is where Nathaniel wrote the def get.

warning: this clause cannot match because  a previous clause at line 2 always matches
  lib/miskatonic/yithians.ex:8

mix compile

We can use defoverridable to any function defined above in a quote block as overridden if the outer scope defines the same name and arity, instead of the outer scope appending clauses to the same name and arity. Although mixing clauses from quote blocks and the outer scope is allowed, it’s mostly going to cause confusing bugs, so I recommend always marking any functions defined in a quote block.

defoverridable No Yes
quote clauses quote clauses quote clauses
defmodule clauses Both defmodule clauses

So, Nathaniel marks get/1 as overridable, and the override works without warnings.

defmodule Miskatonic.Species do
  @callback client_start_link() ::
              {:ok, pid} | {:error, reason :: term}

  defmacro __using__([]) do
    quote do
      @behaviour Miskatonic.Species

      def get(id), do: Miskatonic.Species.get(__MODULE__, id)

      defoverridable get: 1
    end
  end

  def get(module, id) do
    with {:ok, client_pid} <- module.client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end
end

Miskatonic.Species@defoverridable

But, he’s able to do more, when you override a defoverridable function, you can call the overridden function with super. This allows users of your __using__ macro to not have to look at the implementation of the function they are overriding, which means their code is more likely to continue working if you change implementation details.

defmodule Miskatonic.Yithians do
  use Miskatonic.Species

  def client_start_link(keywords \\ [yithian: "Librarian"]) do
    Miskatonic.Clients.Psychic.start_link(keywords)
  end

  def get(id) do
    case super(id) do
      {:error, :not_found} ->
        with {:ok, pid} <- client_start_link(yithian: "Coleopterous") do
          Miskatonic.Client.show(pid, id)
        end
      found ->
        found
    end
  end
end

Miskatonic.Yithians@defoverridable

Miskatonic University’s library is doing really well, but it still has some slight bugs: every module has a get/1 and it’s overridable, but it’s not a callback. It may seem weird to mark get/1 as a callback, since only client code calls get/1, but if we want to make test mocks, to test code that depends on Miskatonic.Species we really need a get/1 callback. By making get/1 a callback, we can also use the compact form of defoverridable, that takes the name of the behaviour whose callbacks are overridable, instead of listing each function’s name/arity.

defmodule Miskatonic.Species do
  @callback client_start_link() ::
              {:ok, pid} | {:error, reason :: term}

  @callback get(id :: String.t) :: term

  defmacro __using__([]) do
    quote do
      @behaviour Miskatonic.Species

      def get(id), do: Miskatonic.Species.get(__MODULE__, id)

      defoverridable Miskatonic.Species
    end
  end

  def get(module, id) do
    with {:ok, client_pid} <- module.client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end
end

Miskatonic.Species@defoverridable-behaviour

One final check that Elixir 1.5 gives us is @impl. @impl is like @Override in Java, but better.

  1. Mark which functions are implementations of callbacks
  2. Document which behaviour a function is for, which makes finding docs and source easier for readers
  3. Force all other callbacks for the same behaviour to use @impl to maintain consistent documentation.

In Miskatonic.Species, there is only one behaviour, but if it was a stack of behaviours, such as building on top of GenServer, then marking which callbacks are for GenServer and which are for other behaviours can be very helpful.

defmodule Miskatonic.Species do
  @callback client_start_link() ::
              {:ok, pid} | {:error, reason :: term}

  @callback get(id :: String.t) :: term

  defmacro __using__([]) do
    quote do
      @behaviour Miskatonic.Species

      @impl Miskatonic.Species
      def get(id), do: Miskatonic.Species.get(__MODULE__, id)

      defoverridable Miskatonic.Species
    end
  end

  def get(module, id) do
    with {:ok, client_pid} <- module.client_start_link() do
      Miskatonic.Client.show(client_pid, id)
    end
  end
end

Miskatonic.Species@impl

TL;DR

Let’s review Miskatonic University’s finding and thank the graduate students for turning mad, so we don’t have to.

  1. We can use use, which calls __using__, which calls quote to inject default implementations
  2. All defs in the quote block should be declared as @callbacks in the outer module where defmacro __using__ is.
  3. Put @behaviour with the outer module as the behaviour name at the top of quote block
  4. The default functions should be one-liners that call functions with the same name in the outer module with __MODULE__ as a prepended argument.
  5. Mark all default functions with @impl, as it will force other callbacks for the behaviour to also use @impl and double check you got the name and arity right between the @callbacks and implementation in the quote block.
  6. Use that passed in __MODULE__ whenever you need to call another callback from the outer module functions, so that overrides for any callback will always be called. Don’t call other outer module functions directly!
  7. Use defoverridable with the outer module so that you don’t have confusing errors with clauses mixing from the quote block and the use using module.