Metrics are a fundamental pillar of observability. There comes a time when it becomes extremely important to monitor your application’s metrics, either to help diagnose possible problems, analyze how users are interacting with your application, or simply to have visibility into how the resources in your app are allocated.
The good news is that the Elixir and Phoenix ecosystem is well prepared to help you with this kind of observability. Many libraries already emit telemetry events using a consistent interface based on Telemetry and Telemetry Metrics. All you have to do is set up a reporter to start aggregating and processing these data.
In this blog, we’ll show you how to use PromEx together with Prometheus to aggregate your app’s metrics. Then we’ll walk through how to use PromEx to automate your Grafana dashboards and make observability as simple as possible.
In part 2 of this post, we’ll explore creating your own PromEx plugins to gather and visualize the exact metrics you want to see from your app.
Prometheus Metrics
Prometheus is a popular open-source project created to collect and store real-time metrics from applications. As described on its official GitHub page, “It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed”.
Prometheus uses an HTTP pull model to get metrics from the apps you want to monitor. As long as the app exposes an HTTP metric endpoint, Prometheus can query it periodically. These metrics normally include a wide variety of information, like system resources, query times, HTTP response status, latency, and many other aspects relevant to understanding system behavior and performance.
Grafana Dashboards
Grafana is an open-source data visualization and monitoring tool that allows users to query, visualize, and analyze data from various sources. It provides a highly customizable and interactive platform for creating dashboards and displaying real-time metrics, analytics, and alerts.
Why Prometheus Metrics and Grafana Dashboards
By using Prometheus and Grafana together, you have a great combo for monitoring and visualizing your application data. Prometheus is excellent for its efficient data collection and storage, and Grafana is great for interactive and rich data visualization. This combination empowers you to monitor, analyze, and troubleshoot your system effectively, enabling you to make data-driven decisions and optimize performance.
The PromEx Library
PromEx is a library made by Alexander Koutmos that helps you integrate your Elixir application with Prometheus and Grafana. It provides a setup for you to start aggregating and exposing Prometheus metrics and it also uses the Grafana Dashboard API to automate the creation of Grafana dashboards. PromEx is easy to use: as described on its GitHub page, “PromEx provides a ‘Plug-in’ style library where you can easily add new plug-ins to surface metrics so that Prometheus can scrape them.”
For example, by just choosing to use the plugin for Oban (PromEx.Plugins.Oban
), you have the
Oban event metrics available to Prometheus and also have some Grafana
dashboards created for you at the application startup. You can see the list of
available built-in plugins on the project Github page.
Using PromEx, Prometheus, and Grafana
Let’s create a new Phoenix app to illustrate the installation and configuration of PromEx. You can also try this on your existing Phoenix project.
We’ll also set up Prometheus and Grafana in our local environment to see them handle the application metrics. Finally, we’ll set up Prometheus and Grafana in our local environment to see all three systems working together.
First, create your new app.
mix phx.new my_app --install
Then configure your database in config/dev.exs
and run the Ecto command in your project folder:
mix ecto.create
Configure Your Phoenix Application
Add the PromEx dependency to your application:
def deps do
[
{:prom_ex, "~> 1.8.0"}
]
end
Fetch the prom_ex dependency by running the following command in the terminal:
mix deps.get
And run:
mix prom_ex.gen.config --datasource YOUR_PROMETHEUS_DATASOURCE_ID
No need to worry if you don’t have YOUR_PROMETHEUS_DATASOURCE_ID
set yet, as we will be doing this later in Grafana.
So we can temporarily have any value here and easily update it later.
This command will create the module MyApp.PromEx
with the instructions to finish setting up PromEx. It’s very
important to follow the instructions carefully. There are four basic steps:
- Update your configuration (
config.exs
,dev.exs
,prod.exs
,releases.exs
, etc.) with the given PromEx config - Add the
MyApp.PromEx
module to your application supervision tree - Update your
endpoint.ex
file to expose your metrics - Choose which plugins and dashboards you want to use (you don’t need to do this step right now, as we will be revisiting it later)
Once again, follow the instructions written in the newly created file carefully to understand the reasoning behind each step.
Let’s start the Phoenix server:
mix phx.server
If the installation worked, you will already be able to see some metrics by hitting the /metrics
endpoint:
localhost:4000/metrics
Now that your application’s metrics are accessible through an endpoint, the next step involves running and configuring Prometheus to ingest this data.
Running Prometheus
You can run Prometheus one of three ways: using the Homebrew package, installing the official Docker image (see below), or installing it from source by following the “getting started” steps of their official install guide (you’re on your own for this one).
Running Prometheus with Homebrew:
brew install prometheus
Please take note of the installation location, as you will need to update a configuration file accordingly.
If you installed it locally, you’ll have to update the prometheus.yml
file the installation created. Just edit the
file by adding the phoenix_app
job definition to the scrape_configs
section:
- job_name: 'phoenix_app'
Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:4000']
You have to restart your Prometheus instance if it was already running.
brew services restart prometheus
Running Prometheus with Docker
You can also run it with Docker, but regardless of the way you choose, we have to configure Prometheus to scrape your app.
To run it with Docker, first we have to create the prometheus.yml
config file. We’ll use a simple example for this
tutorial that you can test in your localhost:
touch prometheus.yml
Open the newly created file and paste the following settings:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
A scrape configuration containing the endpoints to scrape:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['host.docker.internal:9090']
- job_name: 'phoenix_app'
Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['host.docker.internal:4000']
Once you’ve created the prometheus.yml
file, you can point to it using the following Docker command (make sure you
replace “~/my_app/prometheus.yml” with the correct path of your own prometheus.yml file):
docker run \
-p 9090:9090 \
-v ~/my_app/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus
Note: host.docker.internal
was used in the prometheus.yml
file so the container can connect to a service on the
host. This special DNS name works if you are using Docker Desktop. If you are running Docker for Linux, replace
['host.docker.internal:4000']
with ['localhost:4000']
and run this instead:
docker run --network=host \
-p 9090:9090 \
-v ~/my_app/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus
To verify if Prometheus is running properly, navigate to localhost:9090
in your web browser, and you should then be
presented with this screen:
Ensure that your Phoenix server is running, and then click on “Status” followed by “Targets” (or you can directly navigate to http://localhost:9090/targets).
If the State is UP
for the job phoenix_app
, it means that Prometheus is already running correctly and configured to
scrape the metrics of your application.
Let’s specify a metric from your Phoenix app inside the Prometheus dashboard. Navigate to
http://localhost:9090 again and start typing my_app_prom_ex
inside the search input. You
should see a list of available metrics:
(remember to replace my_app
prefix with your app name)
Choose a metric from the list (try locating: my_app_prom_ex_beam_memory_allocated_bytes
) and click on the “Execute”
button to proceed.
You can also change the tab to Graph:
The beam_memory_allocated_bytes
metric should be accessible as PromEx includes a built-in plugin specifically designed
to capture metrics related to the Erlang Virtual Machine (BEAM). Open the MyApp.PromEx
module again and you should
find it listed among the available plugins (Plugins.Beam
). If it appears as a comment, ensure that you have
uncommented it to make the metric accessible.
def plugins do
[
# PromEx built in plugins
Plugins.Application,
Plugins.Beam
# {Plugins.Phoenix, router: MyAppWeb.Router, endpoint: MyAppWeb.Endpoint},
# Plugins.Ecto,
# Plugins.Oban,
# Plugins.PhoenixLiveView,
# Plugins.Absinthe,
# Plugins.Broadway,
# Add your own PromEx metrics plugins
# MyApp.Users.PromExPlugin
]
end
Let’s uncomment the built-in plugin for Phoenix (Plugins.Phoenix
) and restart the server. Afterward, revisit the
Prometheus dashboard and enter my_app_prom_ex_phoenix
in the search input. Now, you should be able to view metrics
specifically related to Phoenix.
As described on the PromEx GitHub page, “The goal of this project is to provide a “Plug-in” style library where you can easily add new plug-ins to surface metrics so that Prometheus can scrape them.” So you can see that with this design, it becomes very easy to add new plugins to your project.
In addition to the plugins
function, the MyApp.PromEx
module also includes the dashboards
function. Now, let’s
proceed with the installation of Grafana and explore how to enable these dashboards.
Running Grafana:
You can install Grafana locally either by following the official instructions, with Homebrew, or by running it in a Docker container.
Brew
brew install grafana
brew services start grafana
Docker
docker run -p 3000:3000 grafana/grafana-enterprise
You can check if it’s running correctly by going to localhost:3000
(use admin
/admin
as the credentials to login).
Now we can configure our Phoenix project so that PromEx starts communicating with Grafana. Let’s go back to the
configuration file which you configured earlier (config.exs
or dev.exs
), and update the grafana
option:
Replace:
config :my_app, MyApp.PromEx,
...
grafana: :disabled,
...
With:
config :my_app, MyApp.PromEx,
...
grafana: [
host: "http://localhost:3000",
# Authenticate via Basic Auth
username: "admin",
password: "admin",
upload_dashboards_on_start: true
],
...
Remember to put your own credentials if you changed your password when logging into Grafana for the first time. You can check more detailed information about those configs in PromEx.Config.
Restart your Phoenix server and you should be able to see the option we set upload_dashboards_on_start: true
taking
place:
my_app % mix phx.server
Generated my_app app
[info] PromEx.DashboardUploader successfully uploaded my_app/_build/dev/lib/prom_ex/priv/application.json.eex to
Grafana.
[info] PromEx.DashboardUploader successfully uploaded my_app/_build/dev/lib/prom_ex/priv/beam.json.eex to Grafana.
[info] Running MyAppWeb.Endpoint with cowboy 2.10.0 at 127.0.0.1:4000 (http)
As you can see in the logged message, some of the PromEx built-in dashboards have been uploaded to Grafana. To access these newly created dashboards, simply navigate to the Dashboards menu in Grafana by visiting http://localhost:3000/dashboards.
You will see these 2 dashboards created: PromEx Application Dashboard PromEx Beam Dashboard
The reason for this is that currently we only have two dashboards configured in the MyApp.PromEx module.
def dashboards do
[
# PromEx built in Grafana dashboards
{:prom_ex, "application.json"},
{:prom_ex, "beam.json"}
# {:prom_ex, "phoenix.json"},
# {:prom_ex, "ecto.json"},
# {:prom_ex, "oban.json"},
# {:prom_ex, "phoenix_live_view.json"},
# {:prom_ex, "absinthe.json"},
# {:prom_ex, "broadway.json"},
# Add your dashboard definitions here with the format: {:otp_app, "path_in_priv"}
# {:my_app, "/grafana_dashboards/user_metrics.json"}
]
end
To have both Phoenix and Ecto entries listed in the Dashboards menu on Grafana, uncomment their respective entries in the configuration file. Afterward, restart your server, and you will be able to see them in the Dashboards menu in Grafana as well.
This confirms that the communication between your Phoenix app and Grafana is working correctly. However, upon clicking any of the available dashboards, you will observe a page similar to the following:
While the layout of the dashboards has been created in Grafana, the dashboards themselves are not yet populated with data. This is because we have not yet established the communication between Grafana and Prometheus, preventing the dashboards from being populated.
To make this work, all you need to do is configure a data source in Grafana. It’s a simple step that will enable the integration between the two tools: Go to the data sources menu on Grafana (http://localhost:3000/datasources) Click on the “Add data source” button. From the list of available data sources, select “Prometheus”.
For this simple tutorial, all we need to set are the data source name and the URL:
The URL should point to your Prometheus instance. In my case, I’m utilizing http://host.docker.internal:9090/ as I’m running Prometheus and Grafana through Docker Desktop. However, if you have them running locally, you can simply use http://localhost:9090/.
Click on the “Save & Test” button located at the bottom of the page, and you should receive a confirmation message stating “data source is working.”
Now, for the final step, you need to take the name you entered and use it as the datasource_id
in the PromEx
configuration file.
Open the MyApp.PromEx
module and update the datasource_id
:
def dashboard_assigns do
[
datasource_id: "my_prometheus_datasource_name",
default_selected_interval: "30s"
]
end
Restart your Phoenix server, go back to the Dashboards page, and start browsing the available dashboards.
They will be populated with metrics collected by Prometheus 🎉.
Wrapping Up
Now that we have access to the built-in metrics and dashboards from popular libraries, maybe we also want to start exploring building our own plugins to be able to start collecting metrics that are specific and relevant to our application. You can dig into Part 2 to discover more about this!