The recommended way to run etcd for kubernetes is to have your etcd cluster outside of the kubernetes cluster. But you also run Prometheus via the Prometheus Operator to monitor everything about your cluster. So how do you get prometheus to monitor your etcd cluster if it isn’t technically a service in kubernetes? We need 3 ingredients: a secret, a service, to which we attach the endpoints of the nodes, and a service monitor.

Create the Secret, Service and ServiceMonitor

Secret

To allow Prometheus to securely connect to etcd, we need a secret. To create a secret we use the following files, which should already be in our possession

...

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd
    release: prometheus
  name: prometheus-etcd
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 30s
    scheme: https
    tlsConfig:
      ca_file: /etc/prometheus/secrets/<secret_name>/ca.crt
      cert_file: /etc/prometheus/secrets/<secret_name>/apiserver-etcd-client.crt
      key_file: /etc/prometheus/secrets/<secret_name>/apiserver-etcd-client.key
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      k8s-app: etcd

Conclusion

That’s it. Now we just need to apply these files to our cluster. If everything went well, connecting to the Prometheus (in the targets section) and Grafana dashboards, you should see the following