...
Let's try to send our alerts by e-mail. To do this we will use 2 other CRDs: Alertmanager and AlertmanagerConfig. Note the analogy with Prometheus and PrometheusRule. Also in this case we have linked the 2 CRDs through labels (however they can be chosen arbitrarily). Launch a describe on the Alertmanager component
| Code Block |
|---|
| language | bash |
|---|
| title | Alertmanager CRD |
|---|
| collapse | true |
|---|
|
# This is just an excerpt from the output of the "describe" command
$ kubectl describe -n monitoring alertmanager
Spec:
Alertmanager Config Selector:
Match Labels:
Alertmanager: config |
Create and configure the AlertmanagerConfig component for sending e-mails (find more details on the official website), using the file below. As you can see, Gmail is used as mail provider here. It isn’t recommended that you use your personal password for this, so you should create an App Password. To do that, go to Account Settings -> Security -> Signing in to Google -> App password (if you don’t see App password as an option, you probably haven’t set up 2-Step Verification and will need to do that first). Copy the newly-created password in the Secret below, after having encrypted it with the command echo <password> | base64 -w0.
| Code Block |
|---|
| language | yml |
|---|
| title | AlertmanagerConfig.yaml |
|---|
| collapse | true |
|---|
|
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: alert-email
namespace: monitoring
labels:
alertmanager: config
spec:
route:
groupBy: [severity]
receiver: 'notifications'
groupWait: 30s
groupInterval: 5m
repeatInterval: 12h
receivers:
- name: 'notifications'
emailConfigs:
- to: <to@example.com>
from: <from@gmail.com>
smarthost: smtp.gmail.com:587
authUsername: <from@gmail.com>
authIdentity: <from@gmail.com>
authPassword:
name: gmail-pass
key: alertmanager.yaml
sendResolved: true
headers:
- key: From
value: <from@gmail.com>
- key: Subject
value: 'Alertmanager notification'
- key: To
value: <to@example.com>
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: gmail-pass
namespace: monitoring
data:
alertmanager.yaml: <pass_encode_base64> |
After executing an apply of the above file, it is advisable to take a look at the logs of the following Pods, to see if there are any errors during the execution
| Code Block |
|---|
| language | bash |
|---|
| title | logs... |
|---|
| collapse | true |
|---|
|
$ kubectl logs -n monitoring pod/mon-kube-prometheus-stack-operator-bcf97f54f-w8s7j
level=info ts=2021-08-20T10:44:28.791312171Z caller=operator.go:1221 component=prometheusoperator key=monitoring/mon-kube-prometheus-stack-prometheus msg="sync prometheus"
level=info ts=2021-08-20T10:44:28.875362927Z caller=operator.go:742 component=alertmanageroperator key=monitoring/mon-kube-prometheus-stack-alertmanager msg="sync alertmanager"
$ kubectl logs -n monitoring pod/alertmanager-mon-kube-prometheus-stack-alertmanager-0
level=info ts=2021-08-20T10:34:03.964Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2021-08-20T10:34:03.964Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml |