Monitor External Etcd Cluster

The recommended way to run etcd for kubernetes is to have your etcd cluster outside of the kubernetes cluster. But you also run Prometheus via the Prometheus Operator to monitor everything about your cluster. So how do you get prometheus to monitor your etcd cluster if it isn’t technicallya service in kubernetes? We need 3 ingredients: a secret, a service, to which we attach the endpoints of the nodes, and a service monitor.

Secret

Service (with endpoints)

Second, the service that will describe our etcd cluster must be created. Moreover, here were are going to list the endpoints for our etcd servers and then attach them to our service. Change the IP addresses to match the IPs of your etcd servers. The way these endpoints are connected to the service is through the name property of the metadata: this must match the name of the service.

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: etcd
  name: prometheus-etcd
  namespace: monitoring
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: metrics
    port: 2379
    protocol: TCP
  selector: null
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: etcd
  name: prometheus-etcd
  namespace: monitoring
subsets:
- addresses:
  - ip: <HOST_ETCD_0>
  - ip: <HOST_ETCD_1>
  - ip: <HOST_ETCD_2>
  ports:
  - name: metrics
    port: 2379
    protocol: TCP

Service monitor

In order for the prometheus operator to easily discover and start monitoring your etcd cluster, a Service Monitor needs to be created. A Service Monitor is a resource defined by the operator that describes how to find a specified service to scrape, our etcd service for example. It also defines things such as how often to scrape, what port to connect to and additionally in this case a configuration for how to establish TLS connections. The paths for the CA, client cert and key are the paths will will mount this files to inside the container.