Page History

The recommended way to run etcd for kubernetes is to have your etcd cluster outside of the kubernetes cluster. But you also run Prometheus via the Prometheus Operator to monitor everything about your cluster. So how do you get prometheus to monitor your etcd cluster if it isn’t technicallya  a service in kubernetes? We need 3 ingredients: a secret, a service, to which we attach the endpoints of the nodes, and a service monitor.

Secret

To allow Prometheus to securely connect to etcd, we need a secret. To create a secret we use the following files, which should already be in our possession

$ kubectl -n monitoring create secret generic <secret_name> --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/apiserver-etcd-client.crt --from-file=/etc/kubernetes/pki/apiserver-etcd-client.key

The mentioned files will be mounted inside the prometheus-0 pod, in the path /etc/prometheus/secrets/<secret_name>.

Service (with endpoints)

Second, the service that will describe our etcd cluster must be created. Moreover, here were are going to list the endpoints for our etcd servers and then attach them to our service. Change the IP addresses to match the IPs of your etcd servers. The way these endpoints are connected to the service is through the name property of the metadata: this must match the name of the service.

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: etcd
  name: prometheus-etcd
  namespace: monitoring
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: metrics
    port: 2379
    protocol: TCP
  selector: null
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: etcd
  name: prometheus-etcd
  namespace: monitoring
subsets:
- addresses:
  - ip: <HOST_ETCD_0>
  - ip: <HOST_ETCD_1>
  - ip: <HOST_ETCD_2>
  ports:
  - name: metrics
    port: 2379
    protocol: TCP

...

ServiceMonitor

In order for the prometheus operator to easily discover and start monitoring your etcd cluster, a Service Monitor ServiceMonitor needs to be created. A Service Monitor ServiceMonitor is a resource defined by the operator that describes how to find a specified service to scrape, our etcd service for example. It also defines things such as how often to scrape, what port to connect to and additionally in this case a configuration for how to establish TLS connections. The paths for the CA, client cert and key are the paths will will mount this files to inside the container.are the paths where the files were mounted within the container.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd
    release: prometheus
  name: prometheus-etcd
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 30s
    scheme: https
    tlsConfig:
      ca_file: /etc/prometheus/secrets/prometheus-etcd/ca.crt
      cert_file: /etc/prometheus/secrets/prometheus-etcd/apiserver-etcd-client.crt
      key_file: /etc/prometheus/secrets/prometheus-etcd/apiserver-etcd-client.key
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      k8s-app: etcd