...
- is our application working?
- how many requests per second is it responding to?
- what are the response times?
- how much network traffic is it producing?
- how stressed are the servers on which our application is located?
- what is the http request that always responds in a very long time?
- the database is unable to respond quickly enough, but maybe there is a bottleneck somewhere?
The Prometheus opensource monitoring solution (official page) can solution can answer these and many other questions and addresses and solves these problems thanks also to the excellent travel companion Grafana. Grafana is a web application that creates graphs divided into panels, with data coming from a variety of different sources, such as OpenTSDB, InfluxDB, ElasticSearc and Prometheus itself.
Installation
...
with Helm
Probably the fastest and most efficient way to get Prometheus is via Helm chart. Add the repo and install the chart (here we work in namespace monitoring
)
...
We present a procedure that establishes the service using Docker-compose. Obviously, Docker-compose must be present on the system (if not present install docker-compose). Create a folder (e.g. "mkdir prometheus") in which we insert the docker-compose.yml
file
Code Block |
---|
language | ymlbash |
---|
title | docker-compose.ymlInstall chart |
---|
collapse | true |
---|
|
version: '3'
services:
prometheus:
image: prom/prometheus
container_name: prometheus
ports:
- 90:9090
restart: always
user: '1000'
volumes:
- "$PWD/promdata:/prometheus"
- "$PWD/promconf:/etc/prometheus:ro"
command: "--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.retention=90d"
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10" |
Always inside the prometheus folder we create 2 other folders, called promconf
and promdata
, where we will insert, respectively, our configurations, present in the prometheus.yml
file, and storage. The latter allows you to configure Prometheus to monitor itself. The just mentioned configuration file is
Code Block |
---|
language | yml |
---|
title | prometheus.yml |
---|
collapse | true |
---|
|
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090'] |
Prometheus collects metrics of monitored targets by scraping the HTTP endpoints of these targets. Since Prometheus himself exposes his internal metrics through the same mechanism, it is possible to scrape and monitor his health through the same mechanism. Now let's launch the service in background mode with the command
Code Block |
---|
language | bash |
---|
title | Launch Prometheus |
---|
|
$ docker-compose up -d
# To check the logs
$ docker-compose logs |
At this point, we open the browser at the address <master_FIP>:<port>
. The service is exposed by default on port 9090, but we have opted for port 90, which must be opened on OpenStack, for security reasons: the port is accessible with the network of the CNAF headquarters or via VPN if you are away.
In general, if we wanted to launch Prometheus with a custom version, we can further modify the prometheus.yml
file. For example, it is possible to modify the global configuration of the Prometheus server, specify the location of additional .yaml
files containing rules that we want to upload to the server or define which resources should be monitored. An extensive overview of the possible configurations is available here.
...
# Add the prometheus-community repo and perform a general update of the repositories
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
# Use the --create-namespace flag, if the namespace does not exist
$ helm install <chart_name> prometheus-community/kube-prometheus-stack -n monitoring [--create-namespace] |
In this way we will have already deployed all the components in our cluster. To perform a quick test, you can connect, via browser, to the user interfaces of Prometheus and Grafana, modifying the two services
in a similar way to what was seen for the Kubernetes dashboard: edit
the type of service
, from ClusterIP
to NodePort
, and select a port in the range 30000-32767. At the first access to Grafana you will be asked for your credentials, which you can later change. Credentials are present in github site.
Section |
---|
Column |
---|
| Image AddedGrafana UI
|
Column |
---|
| Image AddedPrometheus UI
|
|
Upgrade or uninstall chart
To upgrade all the Kubernetes components associated with the chart use or to remove them and delete the release
Code Block |
---|
language | bash |
---|
title | Upgrade or uninstall |
---|
collapse | true |
---|
|
$ helm upgrade <chart_name> prometheus-community/kube-prometheus-stack -n monitoring
$ helm uninstall <chart_name> -n monitoring |
Custom Resource Definitions (CRDs) created by this chart are not removed by default and should be manually cleaned up
Code Block |
---|
language | bash |
---|
title | Remove CRDs |
---|
collapse | true |
---|
|
kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com |
Info |
---|
title | Custom Resource Definitions |
---|
|
The CRD API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource. The name of a CRD object must be a valid DNS subdomain name. |