Announcing Distillery 2.0

It has been several months since my last update, which covered my plans for where to take Distillery in 2.0, and how it would pave the way for releases in Elixir core. It’s been longer than expected, but I’m excited to finally show the community where we’ve landed, and where things are going next.

The 2.0 release is already on Hex, and there is an upgrade guide in the README. The documentation has also been significantly revamped, with improvements across the board, along with new guides that not only cover Distillery features, but deployment in general as well. If you want to take a look at the CHANGELOG, you can find that here.

Here’s a quick overview of the major changes:

• A solution to the problem of runtime configuration
• Improved experience around hot upgrades/downgrades, namely better support for custom appups and programmatically modifying them
• Out of the box support for generating PID files
• Better and more consistent primitives for custom commands and hooks
• Improved errors and better feedback from the CLI
• Major improvements to the documentation: new guides, better organization, searchable docs, and more

Let’s take a closer look at each.

Config Providers

By far the most significant issue for new and veteran users alike, is how to deal with runtime configuration. A properly designed application can be easily configured at runtime - but the place where this falls apart, and the most significant pain point so far, is dealing with dependencies. Specifically, dependencies which expect configuration to be provided when they boot, primarily via the application environment.

Since your own application doesn’t have the opportunity to run any code before those dependencies boot, you don’t have the chance to inject any runtime configuration. The result is that you have to use one of a variety of workarounds to this problem: set REPLACE_OS_VARS to inject values from the system environment; add the dependency to included_applications to take over the lifecycle of a dependency so you can configure it before it is started; or, if available, make use of an init callback provided by the dependency, giving you the opportunity to modify the initial configuration of those libraries during their startup.

Config providers are designed to solve this problem, by providing an abstraction for provisioning configuration before a release is booted, so that all applications in the release can be fully configured at runtime without any special workarounds.

What are config providers?

Config providers themselves are just Elixir modules which implement the Mix.Releases.Config.Provider behaviour, composed of an init/1 callback, which receives a list of arguments as provided in your release configuration file. When invoked, the provider loads configuration from some source, and then pushes the resulting config into the application environment.

You can stack config providers, in order to provide both fallbacks and overrides for configuration values. Config providers are executed in the order they are defined in your release config file, and whichever runs last takes precedence.

How do I use them?

To understand how they are used in a release, let’s consider an example. Distillery provides a Mix.Config config provider out of the box. To use it, you need to configure your release as shown below:

release :myapp do
  set version: current_version(:myapp)
  
  set config_providers: [
    {Mix.Releases.Config.Providers.Elixir, ["${RELEASE_ROOT_DIR}/etc/config.exs"]}
  ]
  
  set overlays: [
    {:copy, "rel/config/config.exs", "etc/config.exs"}
  ]
end

This configuration sets up the Mix.Releases.Config.Providers.Elixir provider for use in the release. It takes a single argument, the path of the config file it should load at runtime. The path can contain references to system environment variables (such as RELEASE_ROOT_DIR as shown), which are expanded at runtime when this provider runs.

NOTE: The environment variable expansion is made available as a helper function as part of the Mix.Releases.Config.Provider module, but must be used by provider implementations taking paths as arguments. It is not related to the REPLACE_OS_VARS functionality, though the syntax is the same - it works whether that setting is active or not.

We ensure the config file is present at the given location with an overlay, which copies rel/config/config.exs into the etc/config.exs, which is a path relative to the root of the release.

NOTE: It is not necessary to include the final config in the release itself - you may want to provision that in a separate step, but the Elixir provider will expect to find a config file at that path, even if it is empty. Other providers may treat the lack of a config differently, but it is recommended that they fail if given a path that doesn’t exist, so that configuration issues prevent the app from booting.

Here’s what that config.exs file might look like for a typical Phoenix app:

use Mix.Config

config :myapp, MyApp.Repo,
  username: System.get_env("DATABASE_USER"),
  password: System.get_env("DATABASE_PASS"),
  database: System.get_env("DATABASE_NAME"),
  hostname: System.get_env("DATABASE_HOST")

port = String.to_integer(System.get_env("PORT"))
config :myapp, MyAppWeb.Endpoint,
  http: [port: port],
  url: [host: "localhost", port: port],
  root: ".",
  secret_key_base: System.get_env("SECRET_KEY_BASE")

Notice how we’re only setting things which need to change at runtime. You can still use all of the config files under config/ in your project, but you should use those for compile-time config and default values only. It is recommended to keep your release configs separated from runtime config by putting the latter under rel/config, to better emphasize the compile-time/runtime distinction.

How they work

Config providers are executed, in the order specified, before a release is booted in a pre-boot configuration phase. Providers are run in their own instance of the VM, which is then terminated once the configuration has been provisioned and persisted to sys.config for the release itself. This environment has all applications in the release loaded, so they are available, but only the :kernel, :stdlib, :compiler, and :elixir applications started.

This dedicated environment allows providers to configure apps like :kernel - which contains some useful runtime config settings, but starts in an environment too restrictive for providers to run in the actual release. In addition, it allows providers to start applications to assist in loading the configuration, e.g. :inets, :crypto, :ssl, etc.

NOTE: Even though you can start other applications in a config provider, if those applications themselves require configuration to run, you must perform that configuration yourself in the provider before starting them.

Once all of the config providers have run, and have pushed their configuration into the application environment (e.g. with Application.put_env/3), Distillery will persist the entire application environment to sys.config, then terminate the configuration-phase VM. At this point, the “real” release VM is booted, using the generated sys.config to fully configure the system at boot.

One consequence of this design is that you cannot store function captures in the config. This should already be familiar to you if you have used releases before, but if you are not, the recommended approach to such config values is to store them as {module, function, args} (MFA) tuples, rather than capturing a function.

This is for two reasons: first, function captures are specific to a particular version of the code when captured - meaning that after an upgrade, that capture is no longer valid, and will result in a :badfun error being raised; second, when a hot upgrade occurs, function captures always refer to the version of the code when they were captured, so code which gets upgraded and then invokes the captured function expecting it to refer to the latest version of the code may fail because it gets results in a different format than expected. Additionally, once old code is purged from the system, the captures will reference non-existent code, and will again raise :badfun when called.

The use of MFA tuples ensures that they can be serialized, and will always reference the current version of the code when invoked - design your configuration to use them instead if needed.

Another consequence of the config provider design is that your config is reified once, at boot, and is then static from that point forward (sans hot upgrades, which is the one case where configs are re-evaluated, as a standard part of the upgrade process).

Your application should be designed to receive configuration at boot, read it from the application env, and then pass it down your supervisor tree, rather than reading directly from the application env when needed. There is nothing enforcing this rule, but config providers are specifically designed with this approach in mind, and are not intended to be used to fetch configuration dynamically once the release has booted.

Further reading

In the Distillery docs, there are sections dedicated to handling runtime configuration and config providers specifically, with an example provider based on JSON files.

You can also find guidance on configuration in general there, particularly examples on architecting your application to push configuration down your supervisor tree from the root supervisor, rather than reading directly from app env.

If you are interested in an example of a non-Elixir config file format, I recently built a TOML library for Elixir, toml, which includes a config provider for TOML files.

NOTE: If you are a current user of my Conform library, I’m going to be deprecating it in favor of toml, as TOML is a formalized spec of the init-style config format that Conform was based around, and with config providers now in Distillery, virtually everything you could do with Conform can now be done with TOML.

Hot Upgrades/Downgrades

While config providers are the “big deal” in this release, there are a number of general quality-of-life improvements which I’m happy to introduce as well. Some of those are specifically oriented around hot upgrades, namely the generation and manipulation of appup files.

A new Mix task - release.gen.appup

The first of these improvements is the addition of a new Mix task, release.gen.appup, which allows you to generate an appup file for a given application and version pair. This file is the same one that Distillery would generate on the fly, but is output to rel/appups/<app>/<v1>_to_<v2>.appup.

Distillery will look in rel/appups for appups describing a particular application and version pair, and if it finds a match, it will use that appup rather than generate a new one. This allows you to modify those appups as needed to better suit the application being upgraded.

This has long been a pain point with working with appups in Exrm and Distillery, as you had to manually place appup files in the right location before, which was not obvious, and prone to accidental deletion. Hopefully this new task will enable a much more pleasant hot upgrade experience!

Appup transforms

The other major improvement in this area is the introduction of another form of extensibility, appup transforms. These are plugins which are designed to transform the appup instruction set programmatically, rather than by hand.

Appup transforms can be used to extend the instruction set, modify the instructions used, remove or replace instructions - essentially whatever transformation you would like to perform is supported.

It turns out Edeliver actually has a similar feature, relup patching, which is similar, but functions a bit differently. My understanding is that the existing “relup patching” plugins will be ported to appup transforms, but if you are interested in what kind of things you might do with them, the link above has a list of transforms that are already used in practice today.

Here’s the signature of an appup transform, which is a no-op (i.e. it doesn’t actually modify the instruction set, just returns it unmodified).

defmodule MyApp.MyAppupTransform do
  use Mix.Releases.Appup.Transform
  
  def up(_app, _v1, _v2, instructions, _opts) do
    instructions
  end
  
  def down(_app, _v1, _v2, instructions, _opts) do
    instructions
  end
end

And here is how you would use it in a release:

release :myapp do
  set appup_transforms: [
    {MyApp.MyAppupTransform, [] = _opts}
  ]
end

It is my hope that tools like appup transforms will better enable the community to share appups and transformations for their own libraries, as well as others, so that Distillery-generated hot upgrades can be relied upon to do the right thing for even the most complex applications.

PID Files

Another quality-of-life improvement in this release is the ability to create a PID file during boot. A PID file is a file which contains the PID of the executable which generated it. When an application terminates that file is removed, and if it is removed while the application is running, the application terminates. If the application restarts, a new PID is written to the file.

These files are used in particular by system supervisors, such as systemd, to better monitor and interact with the services under their purvey. In general I recommend using foreground mode when running under systemd, but if you prefer to run the service as a daemon via start, and have systemd monitor that, PID file support provides better integration than previous releases.

To turn on PID file generation, either export PIDFILE=path/to/pidfile in the system environment, or set -kernel pidfile "path/to/pidfile" in either vm.args or your configuration (we use :kernel to configure it, as the PID file manager is started as a kernel process).

If either of these are set, the PID file manager process will write the current PID to the given path during boot; monitor the file for changes, so that it can terminate the node if the file is deleted, and delete the file during a graceful shutdown.

NOTE: If you terminate a node brutally, the PID file will remain, since our process will never have a chance to execute any cleanup in that case - but shutting down that way is not recommended for a variety of reasons, one of which is the likelihood of resources being left hanging in the wind. If the PID file still exists when the release is restarted, it will be overwritten with the new PID, but it may cause some confusion for external processes like systemd.

Read-Only Mode

It is common to set up a release to run as a particular user, usually with very restrictive permissions. However, you often want to interact with the running release, or execute commands which are part of the release, as a user other than the one it is running as.

Previously, Distillery would always run its same initialization code every time its shell script was invoked - this init code would result in modifying or rewriting files on disk related to the release (primarily sys.config and vm.args). If you ran a command as an elevated user, then tried to start the release, the start would fail, because it could not read, or could not modify, the files it expected it would be able to access, as they had a different owner.

To help alleviate this issue, Distillery allows you to export an environment variable to make the shell scripts operate in a “read-only” mode, RELEASE_READ_ONLY. When set, the scripts will skip writing or modifying any files on disk. The caveat to this, is that you must have run the release at least once without this mode set, so that required files such as vm.args, sys.config, and others, exist for the release to use.

In the future, I hope to eliminate the need for this entirely. The reason why we have this problem is that Distillery takes steps to prevent “original” files in the release from being altered, by copying them to a mutable directory during first boot, leaving the originals immutable.

This was critical when REPLACE_OS_VARS was the primary means of injecting runtime configuration, as once the replacements were performed the first time, the “variables” in the sys.config (or vm.args) are lost, making it impossible for future config changes to take effect. By copying the originals, and then performing the replacement against the copies, we always have the source version to use for variable replacement.

With config providers, we no longer need REPLACE_OS_VARS for sys.config, getting us one step closer to being able to run completely “read-only”, the last obstacle is vm.args, for which we have not implemented an equivalent provider mechanism (or extended config providers to support), yet.

CLI Improvements

A large number of bug fixes and small improvements have been made to the command-line interfaces, particularly with error handling. I won’t get into all of them here, but in general you should expect a smoother, friendlier experience.

NOTE: I consider unfriendly errors or ugly output to be bugs (when they can be fixed), so please open an issue if you encounter any!

RPC and Eval

Two major improvements to the CLI, are the new behaviors for rpc and eval commands. These are breaking changes, but make these commands much more useful and versatile.

Previously, rpc and eval both interacted with a remote node. The rpc command took arguments in the form of rpc <module> <function> [arg1...], while eval took the form eval '<string of code to evaluate>'. In addition to those two, there was also rpcterms <module> <function> '<string representation of argument list>' and command <module> <function>. There were multiple problems with these:

• They are all redundant variations of the same task
• They used Erlang syntax, not Elixir
• It was difficult to remember what syntax to use with which command
• Arguments could be difficult to represent in the shell (proper escaping, etc.) - often leading to frustration just trying to execute simple commands that would take a moment in the Elixir shell

To address this, these commands have been completely reworked in 2.0:

• rpc is used for remote evaluation
• eval is used for local evaluation (i.e. running code in a fresh instance of the VM)
• Both rpc and eval take a string of Elixir code to evaluate, or, can be given --file path/to/script.exs, which will be used as the source of code for evaluation
• Both rpc and eval can also be invoked with --mfa "Module.fun/arity", for invoking functions with arguments provided on the command line. By default this is a 1-to-1 mapping (i.e. the number of args must match the arity), but you can also pass --argv to pass all arguments as a list to the given function (though it must have an arity of 1)
• rpcterms has been removed
• command is deprecated, but has not yet been removed

In addition, rpc and eval both have all of the code in the release available to them without the need for Application.load/1 (so does command, but it will be removed in a future release as it is no longer necessary).

Reworked command handling

Internally, all of the command handling that was previously implemented in escripts is now written in Elixir, and moved into Distillery itself with automated tests - this makes it easier to maintain, and easier to validate. In addition to this rewrite, the UI has been significantly revamped to provide better errors and more useful output.

New command: info

There is a new info command as well. This command only supports one operation currently, info processes, which prints a list of processes in the release - like you would see in Observer - with various options for sorting the output. Run bin/myapp help info processes for more details. I’m hoping to expand the info command in the future to provide more Observer-like capabilities out of the box.

Nicer tools for custom commands

If you have written your own custom commands, then you are probably used to doing something like this:

#!/usr/bin/env bash

"${RELEASE_ROOT_DIR}"/bin/myapp command 'Elixir.MyApp.Tasks' foo -- arg1 #...

This isn’t ideal for a number of reasons, but the main problem is that reinvoking the bin/myapp script means performing a lot of duplicate, unrelated work to set up the environment for that script, when all of that setup has already been performed.

Instead, I have added some helpers for custom commands which should make life much easier, release_ctl, and release_remote_ctl. The latter is an alias of the former, but takes care of passing the required --name and --cookie flags for connecting to the running release.

#!/usr/bin/env bash

# Local evaluation
release_ctl eval 'IO.inspect("hello world!")'
# Local evaluation by applying arguments to the function specified as a flat
# list, i.e. the same way Mix tasks receive arguments
release_ctl eval --mfa 'MyApp.Tasks.run/1' --argv -- "$@"
# Same as above, but rather than passing a list of arguments to the function,
# the number of arguments must match the arity of the function, and arguments
# are applied in those positions. If the number of arguments given does not
# match the arity, then a friendly error is displayed
release_ctl eval --mfa 'MyApp.Tasks.run/2' -- "$1" "$2"
# Remote evaluation versions of the above
release_ctl rpc 'IO.inspect("hello world!")'
release_remote_ctl rpc --mfa 'MyApp.Tasks.run/1' --argv -- "$@"
release_remote_ctl rpc --mfa 'MyApp.Tasks.run/2' -- "$1" "$2"

You can see the full list of available commands and options by running release_ctl with no arguments (from your custom command) or bin/myapp help.

These two helpers avoid re-invoking the main release script, which results in better performance, and easier debugging.

You also have access to erl and elixir helpers which are equivalent to the commands you already know, but run in the release context (i.e. with all the applications in the release loaded). These are more useful for constructing primitives or commands which you want handled by your own CLI implementation.

Documentation Improvements

The docs have been significantly revamped using MkDocs. They are much easier to navigate now, are more readable by extracting tips and warnings into asides/callouts, and are fully searchable as well!

Beyond the surface-level improvements, a variety of new guides have been added, and existing pages have all been reworked, and either merged together or split as needed.

In particular, the following guides have been added:

• Deploying To AWS - this guide walks you through deploying an example application to a fully automated infrastructure in AWS. The infrastructure itself is defined in CloudFormation, following IaC (Infrastructure as Code) principles.
• Working With Docker - this guide walks you through setting up a local Docker environment for building images which can be deployed to a production setting. At the end, you will have a setup which allows you to build and run your app in a Docker container with one command, including with other services like PostgreSQL or Redis.
• Deploying To Digital Ocean - building on Working With Docker, this guide shows you how to easily spin up a containerized environment in Digital Ocean, and run your Docker images there.

I am also working on putting together a guide for Windows, but there is some work yet to be done before that will be ready. It is my hope that these guides will help demystify deployment in Elixir!

For veterans and beginners alike, I would recommend reading Handling Configuration, as it will help you understand in detail the new changes around runtime configuration. This guide also provides tips on how to design your application to be easily configurable and testable.

What’s Next

My primary objective this year was to bring releases into Elixir’s core tooling. During the initial scoping of that work, we realized that configuration was going to be a major obstacle to that effort, because of how deeply intertwined the functionality of releases are with parts of OTP dealing with config.

Distillery 2.0 was planned as a way for the core team and myself to see how a solution for configuration would work with releases - both to gather feedback, as well as be able to lean on the fact that Distillery already exists and has a pool of users who can use their existing expertise and applications to help us determine whether the problems they have are being solved, and whether our solution for configuration is the right one moving forward.

Now that 2.0 is released, my efforts are going to be around gathering that feedback and turning it into a final proposal for the initial implementation of releases in core. Once the core team feels comfortable with that proposal, I’ll begin working on moving the essential release functionality into Elixir itself.

I do have some improvements to Distillery planned, which play into my efforts around the core tools - one of those is building out a mix run equivalent which uses the release primitives to run a Mix project rather than Mix. Basically a mix release.run task which works like mix run --no-halt. I have the design of this sketched out, but I’ve been waiting for this release to have time for implementing it.

I welcome any feedback and suggestions for how to improve these tools, as any such feedback helps me see gaps in my own thinking, as well as give me new ideas for how to address existing issues!

What Isn’t in This Release

I mentioned in my February update that cross-compilation is still a concern - that hasn’t changed, and there are no meaningful advances towards a solution in this release.

I have explored some options, but have not been happy with my experiements so far. I welcome any ideas or suggestions in this area - even better would be people interested in tackling the problem with code! I’m especially interested to see someone experiment with integrating a Nerves-like toolchain approach with Distillery, but beyond Nerves itself, I haven’t seen any efforts in that direction (and have not yet had time myself).

Wrap Up

I hope you’re looking forward to trying out the new release! I’d love to get any feedback, so please feel free to open issues for that purpose - especially if you encounter any issues.

I will be available at ElixirConf and The Big Elixir if you want to talk releases, Distillery, and the future of these tools as well.

DockYard is a digital product agency offering exceptional user experience, design, full stack engineering, web app development, software, Ember, Elixir, and Phoenix services, consulting, and training.