...
How the cluster is upgraded
It can be instructive to analyzeanalyze what happens in the cluster during the update. Then run the update command from the SA and, in another terminal connected to a cluster node, watch live what happens inside it. Run the command
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
$ watch -x kubectl get pod,node -o wide -A #NAMESPACE The following screen will appear, which updatesNAME periodically Every 2.0s: kubectl get pod,node -o wide -A READY STATUS node1: TueRESTARTS Mar 9 17:18:01 2021 NAMESPACE AGE NAME IP NODE NOMINATED NODE READINESS GATES ingress-nginx pod/ingress-nginx-controller-d6vp7 READY STATUS RESTARTS1/1 AGE Running IP 0 NODE default pod/netchecker-agent-hostnet-l6s5x12d 10.233.97.26 master1 1/1 <none> Running 0 <none> kube-system 25hpod/calico-kube-controllers-744ccf69c7-rjwk2 192.168.100.18 1/1 node1 default Running pod/netchecker-agent-hostnet-rf5jl0 1/1 5d21h Running10.233.110.40 0 worker1 <none> 25h 192.168.100.23 node2 default <none> kube-system pod/netcheckercalico-agentnode-hostnet-sc5h74f8n7 1/1 Running 0 1/1 25h 192.168.100.25Running node3 default0 pod/netchecker-agent-kqsz7 12d 192.168.100.206 worker1 <none> 1/1 Running 0 <none> kube-system pod/calico-node-6wkqb 25h 10.233.90.3 node1 default pod/netchecker-agent-lp5pf 1/1 Running 0 1/1 Running 0 12d 25h 10192.233168.92100.294 worker3 node3 default <none> pod/netchecker-agent-z7vb5 <none> kube-system pod/calico-node-9ptsg 1/1 Running 0 25h 1/1 10.233.96.2 Running node2 default0 pod/netchecker-server-f98789d55-xr6n9 12d 1/1 192.168.100.102 Running master2 2 <none> 24h 10.233.96.8 node2 <none> kube-system pod/calico-kube-controllers-596bd759d5-x2zqcnode-b64qh 1/1 Running 0 24h 192.168.100.23 node2 kube-system pod/calico-node-772q21/1 Running 0 12d 1/1 Running192.168.100.190 worker2 0 <none> 25h 192.168.100.23 node2<none> kube-system pod/calico-node-lnh5zlh7p2 1/1 Running 0 25h 192.168.100.25 node3 kube-system12d pod/calico-node-zcqjh 192.168.100.24 master1 <none> <none> kube-system 1/1pod/coredns-645b46f4b6-w4v57 Running 0 25h 192.168.100.181/1 node1 kube-system Running pod/coredns-657959df74-7289c 0 5d21h 1/1 10.233.98.24 Running master2 0 <none> 24h 10.233.96.7 node2<none> kube-system pod/coredns-657959df74-rtl2d645b46f4b6-zmqwk 1/1 Running 0 24h 5d21h 10.233.90.497.30 master1 <none> node1<none> kube-system pod/dns-autoscaler-b5c7869457f7b458498-brq6nkv2k9 1/1 Running 0 24h5d21h 10.233.90.597.28 master1 <none> node1<none> kube-system pod/kube-apiserver-node1master1 1/1 Running 015 12d 25h 192.168.100.1824 master1 node1 kube-system pod/kube-controller-manager-node1<none> <none> kube-system 1/1 pod/kube-apiserver-master2 Running 0 25h 192.168.100.18 node1 kube-system1/1 pod/kube-proxy-67lvh Running 11 (36h ago) 12d 192.168.100.102 master2 <none> 1/1 Running <none> kube-system 0 pod/kube-controller-manager-master1 24h 192.168.100.18 node1 kube-system pod/kube-proxy-whqwb1/1 Running 17 12d 1/1 192.168.100.24 Runningmaster1 0<none> 24h 192.168.100.25 node3 <none> kube-system pod/kube-controller-proxymanager-zs6kfmaster2 1/1 Running 14 (36h ago) 1/1 12d Running 192.168.100.102 0 master2 <none> 24h 192.168.100.23 node2<none> kube-system pod/kube-scheduler-node1proxy-dtsjz 1/1 Running 0 25h5d21h 192.168.100.1894 worker3 <none> node1<none> kube-system pod/metricskube-server-5cd75b7749-d2594proxy-ft984 2/2 Running 0 1/1 24h 10.233.90.6 Running 0 5d21h 192.168.100.206 worker1 <none> node1<none> kube-system pod/nginxkube-proxy-node2hht9g 1/1 Running 0 25h 5d21h 192.168.100.2324 master1 <none> node2<none> kube-system pod/nginxkube-proxy-node3nqbw5 1/1 Running 0 25h 5d21h 192.168.100.25102 node3 kube-systemmaster2 pod/nodelocaldns-hj5t8<none> <none> kube-system pod/kube-proxy-z6mzs 1/1 Running 0 25h 192.168.100.181/1 node1 kube-system Running pod/nodelocaldns-j7zvh 0 5d21h 192.168.100.190 worker2 1/1 <none> Running 0 <none> kube-system 25h 192.168.100.23 node2 kube-system pod/nodelocaldns-jqbx7pod/kube-scheduler-master1 1/1 Running 012 (4d12h ago) 12d 25h 192.168.100.2524 node3 NAMESPACEmaster1 NAME<none> STATUS <none> kube-system ROLES pod/kube-scheduler-master2 AGE VERSION INTERNAL-IP EXTERNAL-IP1/1 OS-IMAGE Running 14 (36h ago) node/node1 Ready control-plane,master 25h v1.20.412d 192.168.100.18102 master2 <none> CentOS Linux 8 <none> kube-system pod/kubernetes-dashboard-5c5f5d4547-mxgmp node/node2 1/1 Ready <none>Running 0 25h v1.20.45d21h 19210.168233.100103.2330 <none> worker2 <none> CentOS Linux 8 <none> kube-system node/node3 Readypod/kubernetes-metrics-scraper-756f68fffd-dqcmk 1/1 <none> Running 0 25h v1.20.45d21h 19210.168233.100.25110.39 worker1 <none> <none> kube-system CentOS Linux 8 |
...
pod/metrics-server-69d9447b96-td7z7 1/1 Running 0 5d21h 10.233.103.31 worker2 <none> <none>
kube-system pod/nginx-proxy-worker1 1/1 Running 0 12d 192.168.100.206 worker1 <none> <none>
kube-system pod/nginx-proxy-worker2 1/1 Running 0 12d 192.168.100.190 worker2 <none> <none>
kube-system pod/nginx-proxy-worker3 1/1 Running 0 12d 192.168.100.94 worker3 <none> <none>
kube-system pod/nodelocaldns-64678 1/1 Running 6 (9d ago) 12d 192.168.100.190 worker2 <none> <none>
kube-system pod/nodelocaldns-d6c5r 1/1 Running 0 12d 192.168.100.94 worker3 <none> <none>
kube-system pod/nodelocaldns-l6ms9 1/1 Running 2 (9d ago) 12d 192.168.100.102 master2 <none> <none>
kube-system pod/nodelocaldns-q2dm9 1/1 Running 1 (12d ago) 12d 192.168.100.206 worker1 <none> <none>
kube-system pod/nodelocaldns-r9m9q 1/1 Running 0 12d 192.168.100.24 master1 <none> <none>
NAMESPACE NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/master1 Ready control-plane,master 252d v1.26.5 192.168.100.24 <none> Rocky Linux 8.8 (Green Obsidian) 4.18.0-477.27.1.el8_8.x86_64 containerd://1.7.1
node/master2 Ready control-plane,master 252d v1.26.5 192.168.100.102 <none> Rocky Linux 8.8 (Green Obsidian) 4.18.0-477.27.1.el8_8.x86_64 containerd://1.7.1
node/worker1 Ready worker 252d v1.26.5 192.168.100.206 <none> Rocky Linux 8.8 (Green Obsidian) 4.18.0-477.27.1.el8_8.x86_64 containerd://1.7.1
node/worker2 Ready worker 252d v1.26.5 192.168.100.190 <none> Rocky Linux 8.8 (Green Obsidian) 4.18.0-477.27.1.el8_8.x86_64 containerd://1.7.1
node/worker3 Ready worker 252d v1.26.5 192.168.100.94 <none> Rocky Linux 8.8 (Green Obsidian) 4.18.0-477.27.1.el8_8.x86_64 containerd://1.7.1 |
The nodes are not updated at the same time, but in turn. The node being updated changes its STATUS to Ready, SchedulingDisabled. As long as it remains in this state, you will notice that all the Pods implemented on it are eliminated and moved to the other available nodes (i.e. the node is in Drain state). Once the update is finished, it will return to Ready and move on to the next node.
Ansible tags
There is a quick way to update only a single aspect of our cluster. Thanks to the tags, we can launch the playbook cluster.yml, which will only update a specific part of the configuration. Let's suppose we want to change the configuration of the ingress, present in the addons.yml file. We make our modification and then, instead of running the playbook upgrade-cluster.yml, we use the command
| Code Block | ||||
|---|---|---|---|---|
| ||||
$ ansible-playbook cluster.yml --tags ingress-controller |
With the --skip-tags flag, instead, it is possible to skip processes. In this example, there is a command to filter and apply only DNS configuration tasks and skip everything else related to host OS configuration and downloading images of containers
| Code Block | ||||
|---|---|---|---|---|
| ||||
$ ansible-playbook cluster.yml --tags preinstall,facts --skip-tags=download,bootstrap-os |
This significantly reduces processing times. The complete list of tags defined in the playbooks can be found here.
| Note | ||
|---|---|---|
| ||
Use |
...
. |