While Prometheus is a great tool for monitoring applications deployed in more traditional ways, at Movio we’re big fans of Docker and our Docker deployment strategy is continuing to evolve. In this third post in our Prometheus series I will look at how our monitoring tools integrate with Kubernetes.
Currently, we’re using Kubernetes to deploy and dynamically schedule our containers and once a system such as this has been deployed statically configuring Prometheus with a list of targets breaks down; having some sort of service discovery mechanism available in your monitoring infrastructure is no longer merely a luxury but a necessity.
Fortunately, Prometheus supports several service discovery options one of them being integration with Kubernetes' API server. Service discovery via the Kubernetes API is currently still considered experimental but has already proven to be quite reliable during our testing and many of Kubernetes’ components support Prometheus natively. Prometheus’ ability to automatically discover services and the way it can utilise Kubernetes’ labels and annotations makes it a powerful tool for monitoring more dynamic infrastructure. Configuring Prometheus to access the Kubernetes API server can be achieved as shown below. A more complete example can be found in the Prometheus documentation. To filter out services which are not appropriately instrumented or that have metrics endpoints exposed under a different path or port we use Prometheus’ relabelling functionality.
Example Prometheus Configuration
...
# Kubernetes scrape targets.
- job_name:'kubernetes'
kubernetes_sd_configs:
-
# The API server addresses. In a cluster this will normally be
# `https://kubernetes.default.svc`. Supports multiple HA API servers.
api_servers:
- https://kubernetes.default.svc
# Run in cluster. This will use the automounted CA certificate and bearer
# token file at /var/run/secrets/kubernetes.io/serviceaccount/ in the pod.
in_cluster: true
# Optional HTTP basic authentication information.
basic_auth:
username: prometheus
password: secret
# Retry interval between watches if they disconnect.
retry_interval:5s
- job_name:'kubernetes-service-endpoints'
kubernetes_sd_configs:
-
api_servers:
- https://kube-master.company.com
in_cluster: true
relabel_configs:
- source_labels:[__meta_kubernetes_role,__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: endpoint;true
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
- source_labels: [__address__,__meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)(?::\d+);(\d+)
replacement: $1:$2
With Prometheus configured we can then use the appropriate annotations in a Kubernetes service definition to control the behaviour of Prometheus. The configuration shown below is part of a service definition that indicates the service we’re deploying into Kubernetes should be scraped but its metrics endpoint is exposed under a non-standard path.
Example Kubernetes service definition:
...
# The service being deployed exposes a metrics endpoint and should be
# scraped by Prometheus but metrics are exposed at a non-standard path and port.
prometheus.io/scrape:'true'
prometheus.io/path: /foo/bar/metrics
prometheus.io/port: '8080'
...
Once Prometheus has discovered a service (scrape target) the target can be scheduled with the knowledge that new instances of the service will be automatically monitored. Being able to deploy or scale a service with the knowledge that it will be automatically monitored is extremely reassuring. Being able to utilise Kubernetes’ labels and annotations to automatically configure Prometheus scrape targets is great but the integration between the two runs even deeper as labels from Kubernetes can also be used within Prometheus’ query language.
Utilising Kubernetes Labels in Queries
As we’ve seen both Prometheus and Kubernetes support the concept of labels and they compliment each other well. The example Prometheus queries thus far have only used labels that were exported directly by the metrics endpoint being scraped (standard Prometheus metric labels) but the labels associated with a Kubernetes service can be combined with the metric labels exposed by scrape targets and can then also be used within Prometheus’ query language.
An example of when this may be useful is a situation where multiple teams or squads are using the same Kubernetes cluster to deploy their services. Multiple teams may deploy a pod or service that exposes the same set of metrics, such as an Apache container that exposes information about the number of requests served, but it may be useful to differentiate between squads when writing queries. In this situation you might expect that you’d have to build several slightly modified versions of a container, one for each squad or parameterise the container with an environment variable to modify the exporter’s behaviour to enable queries that relate to a particular squad. Fortunately neither of these options are necessary. Assuming each squad is using their own Kubernetes namespace the following Prometheus configuration could be used to add a “squad” label to each metric scraped.
Prometheus Configuration:
- source_labels:[__meta_kubernetes_service_namespace]
action: replace
target_label: squad
Metrics can then be easily filtered when executing a query using the “squad” label just as you would with any other label exposed by a scrape target.
Prometheus query:
sum(apache_accesses_total{squad="reporting"})
These labels can contain virtually any information that’s relevant to a service such as a branch name or version number and are extremely useful especially when using third party exporters that you may not necessary want to modify.
Summary
There are many other features and components of Prometheus that have not covered here including recording rules and integration with other products. We’re really excited about Prometheus and the insight it provides into our services and infrastructure as well as the way it integrates with Kubernetes. We’ll certainly be making use of Prometheus more and more as we move forward and would definitely encourage others to experiment and discover the capabilities of this fantastic tool.
Read the other blogs in this series - Prometheus: Lighting the way and Prometheus: Wielding the flame.
About the Author: Mario Weigel
Previously systems administrator at Movio.