Installation with Docker-compose (TLS enabled)
To get an ESK up and running in Docker with security enabled, create the following compose and configuration files:
instances.yml identifies the instances you need to create certificates for;.env sets environment variables to specify the ElasticSearch version and the location where the ElasticSearch certificates will be created;create-certs.yml is a Docker-compose file that launches a container to generate the certificates for ESK;elastic-docker-tls.yml is a Docker-compose file that brings up a ESK instance with Transport Layer Security (TLS) enabled.
instances:
- name: es01
dns:
- es01
- localhost
ip:
- 127.0.0.1
- name: kib01
dns:
- kib01
- localhost
COMPOSE_PROJECT_NAME=es
CERTS_DIR=/usr/share/elasticsearch/config/certificates
VERSION=7.16.3
version: '2.2'
services:
create_certs:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: create_certs
command: >
bash -c '
yum install -y -q -e 0 unzip;
if [[ ! -f /certs/bundle.zip ]]; then
bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
unzip /certs/bundle.zip -d /certs;
fi;
chown -R 1000:0 /certs
'
working_dir: /usr/share/elasticsearch
volumes:
- certs:/certs
- .:/usr/share/elasticsearch/config/certificates
networks:
- elastic
volumes:
certs:
driver: local
networks:
elastic:
driver: bridge
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial # Generate and apply a trial license that supports TLS
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true # Enable TLS to encrypt client communications
- xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.enabled=true # Enable TLS to encrypt internode communications
- xpack.security.transport.ssl.verification_mode=certificate # Allow the use of self-signed certificates by not requiring hostname verification
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
- certs:$CERTS_DIR
ports:
- 9200:9200
networks:
- elastic
healthcheck:
test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
interval: 30s
timeout: 10s
retries: 5
kib01:
image: docker.elastic.co/kibana/kibana:${VERSION}
container_name: kib01
depends_on: {"es01": {"condition": "service_healthy"}}
ports:
- 443:5601 # <--- Pay attention !!!
environment:
SERVERNAME: localhost
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: https://es01:9200
ELASTICSEARCH_USERNAME: kibana_system
ELASTICSEARCH_PASSWORD: CHANGEME # <--- Pay attention !!!
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
SERVER_SSL_ENABLED: "true"
SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
volumes:
- certs:$CERTS_DIR
networks:
- elastic
volumes:
data01:
driver: local
certs:
driver: local
networks:
elastic:
driver: bridge
Generate certificates for ElasticSearch by bringing up the create-certs container, then bring up the ESK
$ docker-compose -f create-certs.yml run --rm create_certs
Creating es_create_certs_run ... done
Archive: /certs/bundle.zip
creating: /certs/ca/
inflating: /certs/ca/ca.crt
creating: /certs/es01/
inflating: /certs/es01/es01.crt
inflating: /certs/es01/es01.key
creating: /certs/kib01/
inflating: /certs/kib01/kib01.crt
inflating: /certs/kib01/kib01.key
$ docker-compose -f elastic-docker-tls.yml up -d
Creating es01 ... done
Creating kib01 ... done
Run the elasticsearch-setup-passwords tool to generate passwords for all built-in users, including the kibana_system user
$ docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords auto --batch --url https://es01:9200"
Changed password for user apm_system
PASSWORD apm_system = <Password>
Changed password for user kibana_system
PASSWORD kibana_system = <Password>
Changed password for user kibana
PASSWORD kibana = <Password>
Changed password for user logstash_system
PASSWORD logstash_system = <Password>
Changed password for user beats_system
PASSWORD beats_system = <Password>
Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = <Password>
Changed password for user elastic
PASSWORD elastic = <Password>
Make a note of the generated passwords. You must configure the kibana_system user password in the compose file to enable Kibana to connect to ElasticSearch, and you’ll need the password for the elastic superuser to log in to Kibana and submit requests to ElasticSearch. Therefore, set ELASTICSEARCH_PASSWORD in the elastic-docker-tls.yml compose file to the password generated for the kibana_system user.
At this point, use docker-compose to restart the ESK
$ docker-compose -f elastic-docker-tls.yml stop
Stopping kib01 ... done
Stopping es01 ... done
$ docker-compose -f elastic-docker-tls.yml up -d
Starting es01 ... done
Recreating kib01 ... done
After the restart, now we can connect to the Kibana dashboard, using the elastic user credentials, and verify that the service is working (remember to use the https protocol). As already seen, you can tear down the containers, network, and volumes by running docker-compose -f elastic-docker-tls.yml down -v.
Run FileBeat on K8s (TLS enabled)
As seen in the parent paragraph, now we have to take care of sending the logs from the K8s cluster to ESK. The procedure is similar to the one already seen, with some small differences. After downloading the filebeat-kubernetes.yaml, make the following changes
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
protocol: https # <--- Pay attention !!!
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl:
certificate_authorities: |
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUDVhtJ8gwO3EZJpY3/OOpv0FwAu0wDQYJKoZIhvcNAQEL
.
.
a7fWt1++yAAS3OHknl7MV8XlnO/S4a3nJz4AH/o=
-----END CERTIFICATE-----
certificate: |
-----BEGIN CERTIFICATE-----
MIIDQDCCAiigAwIBAgIUTy7I4LGEsEhuY69EqGN61v6cBqEwDQYJKoZIhvcNAQEL
.
.
apL+xAsTg06BunbXKYBVZnxIxTA=
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAtPRUO3tAvzWnyZR81qLMbHiWIOnv4lf4fEhSL0L/Oa1XvB+O
.
.
sMpqfmyHtjnNe9VV4u52NrKIDfcpHigXSAGAf2w/LYm/oPsksRCD
-----END RSA PRIVATE KEY-----
-----------------------------------------------------
env:
- name: ELASTICSEARCH_HOST
value: <IP> # <--- Enter the Floating/Internal IP of the host with elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: <elastic_user_password>
The certificates are located inside the es01 container, in the path indicated in the .env file. Retrieve the ca.crt, es01.crt and es01.key files, present in the ca and es01 folder, respectively.
# To enter the container
$ docker exec -it es01 /bin/bash
# Path of the certificates inside the container
root@0d928f8ebff2:/usr/share/elasticsearch/config/certificates# ls
bundle.zip ca es01 kib01