The importance of logs

Logs are very useful for understanding what happens within the cluster, from debugging to monitoring activities. Sometimes, however, the infrastructure where the applications run does not have native tools that offer exhaustive logs. For example, if the container, pod, or node stopped working or were deleted, we would also lose our logs. Therefore, it is advisable to store the logs separately, inside or outside the cluster, to understand the source of the problem or to reuse them in future analysis. This is called cluster-level-logging. Kubernetes does not natively have this paradigm, so sometimes you have to rely on external software. Here we will use ElasticSearch&Kibana (ESK). Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

The procedure described in this paragraph is performed on a VM, that does not belong to the cluster. The ESK service will receive the logs from the cluster being monitored, which will have to take care to correctly point the target VM that receives its data.

Installation with Docker-compose

For the installation of ESK we will use Docker-compose (more info here). It's better to check that the version of Docker-compose is updated and it's recommended that you create a folder and place the docker-compose.yaml file in it.

docker-compose.yaml 
version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.3  # <--- get the latest version
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  k01:
    image: docker.elastic.co/kibana/kibana:7.16.3 # <--- get the latest version
    container_name: k01 
    environment:
      SERVER_NAME: kibana
      ELASTICSEARCH_HOSTS: http://es01:9200
    ports:
      - 80:5601
    networks:
      - elastic
volumes:
  data01:
    driver: local
networks:
  elastic:
    driver: bridge

Open the ports, indicated in the file, on OpenStack: the port relating to ElasticSearch allows the exchange of data between the cluster to be monitored and the service, the one relating to Kibana allows access to the dashboard. Then launch, inside the folder just created, the command (if the file name is different from docker-compose.yaml, then specify it after the -f option)

Start service
$ docker-compose [-f <file_name>] up -d 
Creating es01 ... done
Creating k01  ... done

The command above starts the background service in the shell (it takes a few seconds to allow processing). We then check that the containers are running using

Verify service
$ docker-compose ps
Name              Command               State                       Ports
----------------------------------------------------------------------------------------------
es01   /bin/tini -- /usr/local/bi ...   Up      0.0.0.0:9200->9200/tcp,:::9200->9200/tcp, 9300/tcp
k01    /bin/tini -- /usr/local/bi ...   Up      0.0.0.0:5601->5601/tcp,:::5601->5601/tcp

or equally with the command

Verify service (alternative method)
$ docker ps | grep elastic
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS   PORTS                                                 NAMES 
119eb7e8b36a   kibana:7.16.3          "/bin/tini -- /usr/l…"   2 minutes ago   Up       0.0.0.0:5601->5601/tcp, :::5601->5601/tcp             k01
e902ffce0b93   elasticsearch:7.16.3   "/bin/tini -- /usr/l…"   2 minutes ago   Up       9300/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp   es01

Finally, we can connect to the address http://<FIP>. If we can see the Kibana dashboard, it means that the procedure carried out so far is correct (it may take a couple of minutes from startup to service activation). However, we are not yet able to view the logs generated by our cluster. In the next paragraph we will create a connection between the cluster and the newly instanced log collection service. Conversely, to temporarily interrupt the execution of containers or to permanently delete them use respectively (remember to run these commands inside the folder where the .yaml file is located)

Stop or remove the service
# The command stops execution. To restart it use docker-compose start
$ docker-compose stop
# Stop and remove containers, networks, volumes and images created by "up"
$ docker-compose down [options]

Run FileBeat on K8s

Let's move on to the cluster now, to direct its logs to the newly created data collection service. Download the .yaml file from the link below (more info here)

Download filebeat
# Look at the version
$ curl -LO https://raw.githubusercontent.com/elastic/beats/7.16/deploy/kubernetes/filebeat-kubernetes.yaml

and modify the lines highlighted by the comments in the following extract: if the cluster and the VM that collects the data share the same subnet, you can also use the internal IP of the VM instead of the FIP; to allow the creation of Pods also on the master, add the lines shown at the bottom.

Changes 
output.elasticsearch:
  hosts:['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']	# <--- Enter the desired port
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
-----------------------------------------------------
env:
  name: ELASTICSEARCH_HOST	
  value: <IP>				# <--- Enter the Floating/Internal IP of the host with elasticsearch
  name: ELASTICSEARCH_PORT 
  value: "9200"				# <--- Enter the port (like above)
  name: ELASTICSEARCH_USERNAME
  value: elastic
  name: ELASTICSEARCH_PASSWORD
  value: changeme
  name: ELASTIC_CLOUD_ID
  value:
  name: ELASTIC_CLOUD_AUTH
  value:
  name: NODE_NAME
-----------------------------------------------------
# this toleration is to have the daemonset runnable on master nodes. Remove it if your masters can't run pods
spec:
  template:
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master       
        effect: NoSchedule

Finally, launch FileBeat as DaemonSet, which ensures that there is an agent for each node of the Kubernetes cluster

Lanch filebeat
$ kubectl create -f filebeat-kubernetes.yaml
configmap/filebeat-config created
daemonset.apps/filebeat created
clusterrolebinding.rbac.authorization.k8s.io/filebeat created
clusterrole.rbac.authorization.k8s.io/filebeat created
serviceaccount/filebeat created

To consult the cluster DaemonSets, run the command

Check DaemonSet
$ kubectl get daemonset -n kube-system
NAMESPACE     NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   calico-node   3         3         3       3            3           kubernetes.io/os=linux   13d
kube-system   filebeat      3         3         3       3            3           <none>                   8m26s
kube-system   kube-proxy    3         3         3       3            3           kubernetes.io/os=linux   13d

To remove the DaemonSet use the command (or via the Kubernetes dashboard)

Remove DaemonSet
$ kubectl delete daemonset <ds> -n <namespace>
daemonset.apps "filebeat" deleted

We have therefore created a DaemonSet according to the configuration present in the .yaml file. A DaemonSet generates a Pod for each VM that makes up the cluster (3 in our case). Each Pod has the task of investigating and collecting the logs of the node in which it is located and to send them to the destination set by us. Below is an example screen of the service in operation (Analytics/Discover). Of course, the mountain of data collected can be reduced through queries, by selecting fields or time frames.

  • No labels