The importance of logs

Logs are very useful for understanding what happens within the cluster, from debugging to monitoring activities. Sometimes, however, the infrastructure where the applications run does not have native tools that offer exhaustive logs. For example, if the container, pod, or node stopped working or were deleted, we would also lose our logs. Therefore, it is advisable to store the logs separately, inside or outside the cluster, to understand the source of the problem or to reuse them in future analysis. This is called cluster-level-logging. Kubernetes does not natively have this paradigm, so sometimes you have to rely on external software. Here we will use ElasticSearch&Kibana. Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

The procedure described in this paragraph is performed on a VM, that does not belong to the cluster. The ElasticSearch&Kibana service will receive the logs from the cluster being monitored, which will have to take care to correctly point the target VM that receives its data.

Service installation

For the installation of ElasticSearch and Kibana we will use Docker-Compose (it is better to check that the version of Docker-Compose is updated). It is recommended that you create a folder and place the docker-compose.yml file in it.

...

Remember to run the docker-compose command inside the folder where the .yaml file is located.

Log deployment with FileBeat

Let's move on to the cluster now, to direct its logs to the newly created data collection service. Download the .yaml file from the link (look at the version of the file in the link)

...