...
To be able to back up a k8s cluster, first we need the executable file etcdctl, downloadable from here (choose the appropriate release). Also in the compressed file are two other executables, etcd and etcdutl, which may come in handy in the future. After that, unpack the archive file (this results in a directory containing the binaries) and add the executable binaries to your path (i.e. /usr/local/bin)
| Code Block |
|---|
| language | bash |
|---|
| title | Download binary |
|---|
|
# For example, let's download release 3.5.4
$ wget https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz
$ tar xzvf etcd-v3.5.4-linux-amd64.tar.gz
# In addition to the etcdctl executable, we also take etcd and etcdutl
$ sudo cp etcd-v3.5.4-linux-amd64/etcdctletcd* /usr/local/bin/
$# etcdctl version
etcdctl version: 3.5.4
API version: 3.5Check that everything is OK
$ etcdctl version
etcdctl version: 3.5.4
API version: 3.5
$ etcdutl version
etcdutl version: 3.5.4
API version: 3.5
$ etcd --version
etcd Version: 3.5.4
Git SHA: 08407ff76
Go Version: go1.16.15
Go OS/Arch: linux/amd64 |
Once we have the executable file, we need the certificates to be able to communicate with the etcd node(s). If you don't know the location of the certificates, you can retrieve it using the grep command in the /etc/kubernetes folder on the master node (the default directory that holds the certificates in the etcd node is /etc/ssl/etcd/ssl). Save the location of the certificates in the following environment variables
| Code Block |
|---|
| language | bash |
|---|
| title | etcd certificates |
|---|
|
# Insert the following lines inside the ".bashrc" file, then use "$ source .bashrc" to apply the changes
export ETCDCTL_CERT=/<path>/cert.pem
export ETCDCTL_CACERT=/<path>/ca.pem
export ETCDCTL_KEY=/<path>/key.pem
ETCDCTL_ENDPOINTS=etcd1:2379,etcd2:2379,etcd3:2379 |
...
| Code Block |
|---|
| language | bash |
|---|
| title | Pause cluster |
|---|
|
# Let's go to the master(s) and temporarily move the "kube-apiserver.yaml" file
$ sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# Stop etcd service on etcd node(s)
$ sudo systemctl stop etcd.service |
...
| Code Block |
|---|
| language | bash |
|---|
| title | Copy snapshot |
|---|
|
# Paste the snapshot into the path where the etcd node data are stored
$ sudo cp -r <path>/<restore> /var/lib/etcd/
# For each etcd node
$ sudo cp -r $HOME/etcd1.etcd/member/ /var/lib/etcd/ |
...
| Code Block |
|---|
| language | bash |
|---|
| title | Restart cluster |
|---|
|
# Start etcd service on etcd node(s)
$ sudo systemctl start etcd.service
# Restore the API server from the master(s)
$ sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/ |
| Tip |
|---|
We It's also recommend restarting any components (e.g. kube-scheduler, kube-controller-manager, kubelet) to ensure that they don't rely on some stale data. Note that in practice, the restore takes a bit of time. During the restoration, critical components will lose leader lock and restart themselves. |
...