When it comes to application and infrastructure monitoring there are a myriad of options to choose from. One option in particular has captured our attention recently at Movio and that is Prometheus, an open source monitoring system and time series database originally developed by SoundCloud which is now a standalone open source project. Prometheus provides a unique approach and feature set that aligns well with our shift towards Docker and a microservices architecture. This is the first in a series of posts in which we’ll explore the basics of deploying Prometheus, getting started with its query language and some of the features that make it an excellent choice for monitoring modern applications deployed using Docker and a new generation of cluster scheduling tools.

Getting Started

Prometheus consists of several components however there are two main components which must be present to get started, the Prometheus server itself and an exporter or appropriately instrumented service that exposes metrics the server can scrape periodically. Prometheus operates via a pull model; endpoints are constantly scraped (polled) at configurable intervals for metrics which are then stored by the server as time series data with each time series being uniquely identified by its metric name and a set of key-value pairs called labels. There are a number of officially and independently maintained exporters available that expose everything from OS level and SNMP metrics through to Apache and JMX metrics. In addition to this, several client libraries are available that application developers can use to instrument their own code and expose custom metrics.

A Prometheus server can be deployed quickly by using the provided Docker container and can actually be configured to monitor itself as a starting point since the server conveniently exposes a metrics endpoint.

docker run \
  --detach=true \
  -p 9090:9090 \
  -v ${PATH_TO_STORAGE}:/prometheus \
  -v ${PATH_TO_CONFIG}:/etc/prometheus/prometheus.yml \
  --name prometheus \
  prom/prometheus

The Prometheus server includes the ability to execute queries, generate graphs and also exposes a status page which can be used to inspect the current configuration and the status of targets being monitored. Once the server is up and running you can begin to explore one of the most powerful features of Prometheus, its query language.

Querying Prometheus

Prometheus provides a powerful functional expression language that enables the user to perform queries on its time series data. Each time series consists of a metric name and set of labels which enable Prometheus’ highly dimensional data model. Let’s take a look at writing some queries to interpret metrics exposed by the node exporter, an exporter for machine metrics which can also be conveniently deployed as a Docker container. To inspect the one-minute load average for a single machine we can use a simple query consisting only of a metric name “node_load1” and single label “instance” that identifies the particular machine we’re interested in.

Query:

node_load1{instance="test-db1.example.com:9100"}

Result:

node_load1{instance="test-db1.example.com:9100",job="node"}  1.35

This query returns a single result, an instant vector, which is the latest value Prometheus has obtained from “test-db1.example.com”. Queries can of course also return a set of results to show how the value has changed over time. Results can be graphed directly in Prometheus but our preferred option at Movio currently is to use Grafana.

When starting out with Prometheus and the node exporter you may find that the values of some metrics bear little resemblance to what you’d typically expect to see. This is because Prometheus exporters generally favour exposing raw metrics; queries to interpret those metrics are written and executed on the server. This is the case with the next example which involves a slightly more complex query used to inspect the iowait value over time for a particular host. Again this metric is exposed by the node exporter. You may begin by executing the following simple query:

Query:

node_cpu{instance="test-db1.example.com:9100",mode="iowait"}

Result:

node_cpu{cpu="cpu5",instance="test-db1.example.com:9100",mode="iowait"}  218543.75
node_cpu{cpu="cpu2",instance="test-db1.example.com:9100",mode="iowait"}  292970.83
node_cpu{cpu="cpu6",instance="test-db1.example.com:9100",mode="iowait"}  217703.81
node_cpu{cpu="cpu3",instance="test-db1.example.com:9100",mode="iowait"}  508336.58
node_cpu{cpu="cpu1",instance="test-db1.example.com:9100",mode="iowait"}  311947.31
node_cpu{cpu="cpu7",instance="test-db1.example.com:9100",mode="iowait"}  26742.46
node_cpu{cpu="cpu4",instance="test-db1.example.com:9100",mode="iowait"}  328981.23
node_cpu{cpu="cpu0",instance="test-db1.example.com:9100",mode="iowait"}  606676.11 

This is not what you’d typically see when using traditional Linux monitoring tools such as iostat where the iowait value would be displayed as a percentage of CPU time. How does the node exporter arrive at these values and what do they mean? Looking at the relevant source code we can see that these values are collected from /proc/stat as they normally would be. But in their raw form these values are just ever increasing counters and provide little insight into how a machine is performing. To make sense of these results we need to delve a little deeper into the query language and start to make use of the functions and operators that Prometheus provides us.

To calculate iowait as a percentage of CPU time we need to inspect the reported values over some period of time not just at a particular instant. To do this we build on our original query  and add a range vector selector combined with the irate() function. The range vector selector selects “a range of samples back from the current instant” and the irate() function “calculates the per-second instant rate of increase of the time series in the range vector”.

Query:

irate(node_cpu{instance="test-db1.example.com:9100",mode="iowait"}[1m])

Result:

{cpu="cpu0",instance="test-db1.example.com:9100",mode="iowait"}  0.038633185
{cpu="cpu5",instance="test-db1.example.com:9100",mode="iowait"}  0.013321788
{cpu="cpu2",instance="test-db1.example.com:9100",mode="iowait"}  0.017984414
{cpu="cpu6",instance="test-db1.example.com:9100",mode="iowait"}  0.012655698
{cpu="cpu3",instance="test-db1.example.com:9100",mode="iowait"}  0.027309665
{cpu="cpu1",instance="test-db1.example.com:9100",mode="iowait"}  0.019982682
{cpu="cpu7",instance="test-db1.example.com:9100",mode="iowait"}  0.004662626
{cpu="cpu4",instance="test-db1.example.com:9100",mode="iowait"}  0.017984414

Typically though we don’t look at the value of iowait per-CPU core (although Prometheus allows us to do this) but as an average across all cores in the system. Prometheus provides several aggregation operators that allow us to combine results. In this case we’ll use an average operator to calculate the average rate of increase across all CPU cores:

Query:

avg(irate(node_cpu{instance="test-db1.example.com:9100",mode="iowait"}[1m]))

Result:

0.019066809

We’re getting closer but this value still looks unfamiliar. In Linux the values in /proc/stat are reported in clock ticks but the node exporter divides each value by user HZ (100) to give each time as a fraction of a second. If we want the percentage that we’re used to dealing with then we need only use an arithmetic operator to multiply by 100.

Query:

avg(irate(node_cpu{instance="test-db1.example.com:9100",mode="iowait"}[1m])) * 100

Result:

1.9066809

This query produces a familiar value which is iowait as a percentage of CPU time averaged across all cores in the system and once graphed provides an insight into IO performance.

Like any new language Prometheus’ expression language can take some getting used to but it’s worth investing some time in learning as it is extremely flexible. Queries can easily be written to aggregate data from multiple hosts and we’ve really only scratched the surface of what can be achieved here.

In summary

Using exporters that are already available, such as the node exporter, is a great way to get started monitoring commonly used infrastructure components and familiarising yourself with the query language but this is just the beginning!

In future posts we’ll examine how the development team at Movio are making use of Prometheus’ client libraries to instrument their own code in order to gain a deeper understanding of application behaviour and performance. We’ll also be taking a look at how Prometheus integrates with Kubernetes to support service discovery and monitoring of dynamically scheduled services.

About the Author: Mario Weigel

 Previously systems administrator at Movio.