The recommended way to run etcd for kubernetes is to have your etcd cluster outside of the kubernetes cluster. But you also run Prometheus via the Prometheus Operator to monitor everything about your cluster. So how do you get prometheus to monitor your etcd cluster if it isn’t technically a service in kubernetes? We need 3 ingredients: a secret, a service, to which we attach the endpoints of the nodes, and a service monitor.

Create the Secret, Service and ServiceMonitor

Secret

To allow Prometheus to securely connect to etcd, we need a secret. To create a secret we use the following files, which should already be in our possession

...

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: etcd
  name: prometheus-etcd
  namespace: monitoring
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: metrics
    port: 2379
    protocol: TCP
  selector: null
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: etcd
  name: prometheus-etcd
  namespace: monitoring
subsets:
- addresses:
  - ip: <HOST_ETCD_0>	# <--- Insert IP
  - ip: <HOST_ETCD_1>	# <--- Insert IP
  - ip: <HOST_ETCD_2>	# <--- Insert IP
  ports:
  - name: metrics
    port: 2379
    protocol: TCP

ServiceMonitor

In order for the prometheus operator to easily discover and start monitoring your etcd cluster, a ServiceMonitor needs to be created. A ServiceMonitor is a resource defined by the operator that describes how to find a specified service to scrape, our etcd service for example. It also defines things such as how often to scrape, what port to connect to and additionally in this case a configuration for how to establish TLS connections. The paths for the CA, client cert and key are the paths where the files were mounted within the container.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd
    release: prometheus
  name: prometheus-etcd
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 30s
    scheme: https
    tlsConfig:
      ca_filecaFile: /etc/prometheus/secrets/<secret_name>/ca.crt
      cert_filecertFile: /etc/prometheus/secrets/<secret_name>/apiserver-etcd-client.crt
      key_filekeyFile: /etc/prometheus/secrets/<secret_name>/apiserver-etcd-client.key
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      k8s-app: etcd

Conclusion

That’s it. Now we just need to apply these files to our cluster. If everything went well, connecting to the Prometheus (in the targets section) and Grafana dashboards, you should see the following

...