To get an ESK up and running in Docker with security enabled, create the following compose and configuration files:
instances.yml identifies the instances you need to create certificates for;.env sets environment variables to specify the ElasticSearch version and the location where the ElasticSearch certificates will be created;create-certs.yml is a Docker-compose file that launches a container to generate the certificates for ESK;elastic-docker-tls.yml is a Docker-compose file that brings up a ESK instance with Transport Layer Security (TLS) enabled.instances:
- name: es01
dns:
- es01
- localhost
ip:
- 127.0.0.1
- name: kib01
dns:
- kib01
- localhost |
COMPOSE_PROJECT_NAME=es CERTS_DIR=/usr/share/elasticsearch/config/certificates VERSION=7.16.3 |
version: '2.2'
services:
create_certs:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: create_certs
command: >
bash -c '
yum install -y -q -e 0 unzip;
if [[ ! -f /certs/bundle.zip ]]; then
bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
unzip /certs/bundle.zip -d /certs;
fi;
chown -R 1000:0 /certs
'
working_dir: /usr/share/elasticsearch
volumes:
- certs:/certs
- .:/usr/share/elasticsearch/config/certificates
networks:
- elastic
volumes:
certs:
driver: local
networks:
elastic:
driver: bridge |
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial # Generate and apply a trial license that supports TLS
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true # Enable TLS to encrypt client communications
- xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.enabled=true # Enable TLS to encrypt internode communications
- xpack.security.transport.ssl.verification_mode=certificate # Allow the use of self-signed certificates by not requiring hostname verification
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
- certs:$CERTS_DIR
ports:
- 9200:9200
networks:
- elastic
healthcheck:
test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
interval: 30s
timeout: 10s
retries: 5
kib01:
image: docker.elastic.co/kibana/kibana:${VERSION}
container_name: kib01
depends_on: {"es01": {"condition": "service_healthy"}}
ports:
- 443:5601 # <--- Pay attention !!!
environment:
SERVERNAME: localhost
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: https://es01:9200
ELASTICSEARCH_USERNAME: kibana_system
ELASTICSEARCH_PASSWORD: CHANGEME # <--- Pay attention !!!
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
SERVER_SSL_ENABLED: "true"
SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
volumes:
- certs:$CERTS_DIR
networks:
- elastic
volumes:
data01:
driver: local
certs:
driver: local
networks:
elastic:
driver: bridge |
Generate certificates for ElasticSearch by bringing up the create-certs container, then bring up the ESK
$ docker-compose -f create-certs.yml run --rm create_certs Creating es_create_certs_run ... done Archive: /certs/bundle.zip creating: /certs/ca/ inflating: /certs/ca/ca.crt creating: /certs/es01/ inflating: /certs/es01/es01.crt inflating: /certs/es01/es01.key creating: /certs/kib01/ inflating: /certs/kib01/kib01.crt inflating: /certs/kib01/kib01.key $ docker-compose -f elastic-docker-tls.yml up -d Creating es01 ... done Creating kib01 ... done |
At this point, Kibana cannot connect to the Elasticsearch cluster. You must generate a password for the built-in |
Run the elasticsearch-setup-passwords tool to generate passwords for all built-in users, including the kibana_system user
$ docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords auto --batch --url https://es01:9200" Changed password for user apm_system PASSWORD apm_system = <Password> Changed password for user kibana_system PASSWORD kibana_system = <Password> Changed password for user kibana PASSWORD kibana = <Password> Changed password for user logstash_system PASSWORD logstash_system = <Password> Changed password for user beats_system PASSWORD beats_system = <Password> Changed password for user remote_monitoring_user PASSWORD remote_monitoring_user = <Password> Changed password for user elastic PASSWORD elastic = <Password> |
Make a note of the generated passwords. You must configure the kibana_system user password in the compose file to enable Kibana to connect to ElasticSearch, and you’ll need the password for the elastic superuser to log in to Kibana and submit requests to ElasticSearch. Therefore, set ELASTICSEARCH_PASSWORD in the elastic-docker-tls.yml compose file to the password generated for the kibana_system user.
At this point, use docker-compose to restart the ESK
$ docker-compose -f elastic-docker-tls.yml stop Stopping kib01 ... done Stopping es01 ... done $ docker-compose -f elastic-docker-tls.yml up -d Starting es01 ... done Recreating kib01 ... done |
After the restart, now we can connect to the Kibana dashboard, using the elastic user credentials, and verify that the service is working (remember to use the https protocol). As already seen, you can tear down the containers, network, and volumes by running docker-compose -f elastic-docker-tls.yml down -v.
As seen in the parent paragraph, now we have to take care of sending the logs from the K8s cluster to ESK. The procedure is similar to the one already seen, with some small differences. After downloading the filebeat-kubernetes.yaml, make the following changes
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
protocol: https # <--- Pay attention !!!
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl:
certificate_authorities: |
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUDVhtJ8gwO3EZJpY3/OOpv0FwAu0wDQYJKoZIhvcNAQEL
.
.
a7fWt1++yAAS3OHknl7MV8XlnO/S4a3nJz4AH/o=
-----END CERTIFICATE-----
certificate: |
-----BEGIN CERTIFICATE-----
MIIDQDCCAiigAwIBAgIUTy7I4LGEsEhuY69EqGN61v6cBqEwDQYJKoZIhvcNAQEL
.
.
apL+xAsTg06BunbXKYBVZnxIxTA=
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAtPRUO3tAvzWnyZR81qLMbHiWIOnv4lf4fEhSL0L/Oa1XvB+O
.
.
sMpqfmyHtjnNe9VV4u52NrKIDfcpHigXSAGAf2w/LYm/oPsksRCD
-----END RSA PRIVATE KEY-----
-----------------------------------------------------
env:
- name: ELASTICSEARCH_HOST
value: <IP> # <--- Enter the Floating/Internal IP of the host with elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: <elastic_user_password> |
The certificates are located inside the es01 container, in the path indicated in the .env file. Retrieve the ca.crt, es01.crt and es01.key files, present in the ca and es01 folder, respectively.
# To enter the container $ docker exec -it es01 /bin/bash # Path of the certificates inside the container root@0d928f8ebff2:/usr/share/elasticsearch/config/certificates# ls bundle.zip ca es01 kib01 |