You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

Alerting rules allow you to define alert conditions based on Prometheus Quey Language (PromQL) expressions and to send notifications about firing alerts to an external service. Whenever the alert expression results in one or more vector elements at a given point in time, the alert counts as active for these elements' label sets. If you have installed Prometheus with Helm, you have yet a Custom Resource Definitions (CRD) called PrometheusRule. Below is an example

PrometheusRule.yaml 
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    meta.helm.sh/release-name: mon
    meta.helm.sh/release-namespace: monitoring
  labels:
    app: kube-prometheus-stack
    release: mon
  name: regolatest
  namespace: monitoring
spec:
  groups:
  - name: customRules
    rules:
    - alert: rateGraph
      expr: rate(prometheus_http_requests_total{handler="/graph"}[5m]) * 60 >= 2
      labels:
        severity: none # You can add other labels, which can be useful in the future for grouping alerts
      annotations:
        description: This alert is activated if, in the last 5 minutes, the http requests per minute towards the sub-path "/graph" of the Prometheus dashboard exceeds the value 2
        summary: Many requests to "graph"
    - alert: rateAlerts
      expr: rate(prometheus_http_requests_total{handler="/alerts"}[5m]) * 60 >= 3
      labels:
        severity: none # You can add other labels, which can be useful in the future for grouping alerts
      annotations:
        description: This alert is activated if, in the last 5 minutes, the http requests per minute towards the sub-path "/alerts" of the Prometheus dashboard exceeds the value 3
        summary: Many requests to "alerts"

In this example we have inserted 2 rules that evaluate the rate of http requests to Prometheus dashboard pages. We will then update these web pages via the browser, to trigger the alert. Also note the labels adopted in this manifest. Similarly to what was seen in Monitor a Service, the labels must match those present in the ruleSelector field.

Prometheus CRD 
# This is just an excerpt from the output of the "describe" command
$ kubectl describe -n monitoring prometheus 
Spec:
  Rule Selector:
    Match Labels:
      App:      kube-prometheus-stack
      Release:  mon

Trigger the rules

Now let's move to the Prometheus dashboard and update the sub-path /graph and /alerts (framed in yellow at the top left of the image) a few times (15-20 times should be enough), in order to activate the rules adopted. As you can see from the image, the created alerts have been activated and, in fact, are in firing status. After a few minutes, if no other http requests are made, these alerts will return to the starting status, i.e. inactive. The Watchdog alert, at the top of the screenshot, is always present: is an alert meant to ensure that the entire alerting pipeline is functional.Firing alerts

Send alert via e-mail

Let's try to send our alerts by e-mail. To do this we will use 2 other CRDs: Alertmanager and AlertmanagerConfig. Note the analogy with Prometheus and PrometheusRule. Also in this case we have linked the 2 CRDs through labels (however they can be chosen arbitrarily). Launch a describe on the Alertmanager component

# This is just an excerpt from the output of the "describe" command
$ kubectl describe -n monitoring alertmanager
Spec:
  Alertmanager Config Selector:
    Match Labels:
      Alertmanager: config

Create and configure the AlertmanagerConfig component for sending e-mails. As you can see, Gmail is used as mail provider here. It isn’t recommended that you use your personal password for this, so you should create an App Password. To do that, go to Account Settings -> Security -> Signing in to Google -> App password (if you don’t see App password as an option, you probably haven’t set up 2-Step Verification and will need to do that first). Copy the newly-created password in the Secret below, after having encrypted it with the command echo <password> | base64 -w0.

AlertmanagerConfig.yaml 
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-email
  namespace: monitoring
  labels:
    alertmanager: config
spec:
  route:
    groupBy: [severity]
    receiver: 'notifications'
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
  receivers:
  - name: 'notifications'
    emailConfigs:
    - to: <to@example.com>
      from: <from@gmail.com>
      smarthost: smtp.gmail.com:587
      authUsername: <from@gmail.com>
      authIdentity: <from@gmail.com>
      authPassword:
        name: gmail-pass
        key: alertmanager.yaml
      sendResolved: true
      headers:
      - key: From
        value: <from@gmail.com>
      - key: Subject
        value: 'Alertmanager notification'
      - key: To
        value: <to@example.com>
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: gmail-pass
  namespace: monitoring
data:
  alertmanager.yaml: <pass_encode_base64>
  • No labels