...
Live migration for instances created on ephemeral storage
This procedure must be used only from/to hypervisor using LVM thin
Let's suppose that we want to migrate a VM with id dad2d364-3432-454d-a72d-910067e0187d from compute-02 to compute-04
It's a good practice to check if there is sufficient space in the LVM before taking snapshots and, anyway, "fstrim" the LVM-thin that mange /var/lib/nova/instances before taking snapshots (see Data% and Meta% columns):
| Code Block | ||
|---|---|---|
| ||
[root@compute-02 instances]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvmnova lvmvg Vwi-aotz-- <1.82t pool00 99.48
pool00 lvmvg twi-aotz-- <1.82t 99.48 24.80
[root@compute-02 instances]# fstrim /var/lib/nova/instances/
[root@compute-02 instances]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvmnova lvmvg Vwi-aotz-- <1.82t pool00 0.16
pool00 lvmvg twi-aotz-- <1.82t 0.16 11.04
|
First of all on the source hypervisor let's take a LV snapshotTo migrate an instance from a compute node to another one, you can use the "openstack server migrate" command with the '–block-migration' option, after having sourced the admin openrc script. E.g.:
| Code Block | ||
|---|---|---|
| ||
openstack server migrate[root@compute-02 ~]# lvcreate -s --live cld-np-14.cloud.pd.infn.it --block-migration e1914906-7e33-4d30-88f0-785130fdd85d |
This will live migrate the instance with uuid e1914906-7e33-4d30-88f0-785130fdd85d to the compute node cld-np-14.cloud.pd.infn.it.
name lvmnova.dad2d364-3432-454d-a72d-910067e0187d lvmvg/lvmnova
WARNING: Sum of all thin volume sizes (335.75 GiB) exceeds the size of thin pool lvmvg/pool00 and the size of whole volume group (167.88 GiB).
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "lvmnova.dad2d364-3432-454d-a72d-910067e0187d" created.
[root@compute-02 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvmnova lvmvg Vwi-aotz-- <167.88g pool00 6.11
lvmnova.dad2d364-3432-454d-a72d-910067e0187d lvmvg Vwi---tz-k <167.88g pool00 lvmnova
pool00 lvmvg twi-aotz-- <167.88g 6.14 11.04
[root@compute-02 ~]# |
Let's set the snapshot in read only mode:
| Code Block | ||
|---|---|---|
| ||
[root@compute-02 ~]# lvchange -p r lvmvg/lvmnova.dad2d364-3432-454d-a72d-910067e0187d
Logical volume lvmvg/lvmnova.dad2d364-3432-454d-a72d-910067e0187d changed.
[root@compute-02 ~]# |
Let's migrate the instance,using the "openstack server migrate" command with the '–block-migration' option, after having sourced the admin openrc script, i.e.In train the "–live" option is deprecated. You can use:
| Code Block | ||
|---|---|---|
| ||
[root@controller-01 ~]# openstack server migrate --os-compute-api-version 2.30 --live-migration --host compute-0304.cloud.pd.infn.it --block-migration dad2d364-3432-os454d-compute-api-version 2.79 92c413cb-d6f2-4103-9042-9e2e85257029a72d-910067e0187d [root@controller-01 ~]# |
The command doesn't work if source and target hypervisor aren't compatible. The error would be something like:
...
In this case, please find another compatible hypervisor, or you can use cold migration (i.e. the VM will be powered off during the migration)
If the block live migration for some reason didn't work and you have a "corrupted VM"
| Code Block |
|---|
[root@controller-01 ~]# nova stop dad2d364-3432-454d-a72d-910067e0187d
[root@controller-01 ~]# |
| Code Block |
|---|
## Activate snapshot
root@compute-02 ~]# lvchange -ay -K lvmvg/lvmnova.dad2d364-3432-454d-a72d-910067e0187d
## Mount snapshot
[root@compute-02 ~]# mkdir /mnt/snap
[root@compute-02 ~]# mount /dev/lvmvg/lvmnova.dad2d364-3432-454d-a72d-910067e0187d /mnt/snap -o ro,nouuid,norecovery
[root@compute-02 ~]# ll /mnt/snap/dad2d364-3432-454d-a72d-910067e0187d/
console.log disk disk.info
## Copy disk image to target hypervisor
[root@compute-02 ~]# scp /mnt/snap/dad2d364-3432-454d-a72d-910067e0187d/disk compute-04.cloud.pd.infn.it:/var/lib/nova/instances/dad2d364-3432-454d-a72d-910067e0187d
The authenticity of host 'compute-04.cloud.pd.infn.it (192.168.60.89)' can't be established.
ED25519 key fingerprint is SHA256:Fqq7jDeH98KxJ3J9QCFnBaWdiESy7gZ9+toHWU8i1jA.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:1: compute-04
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'compute-04.cloud.pd.infn.it' (ED25519) to the list of known hosts.
root@compute-04.cloud.pd.infn.it's password:
disk
[root@compute-02 ~]# umount /mnt/snap
[root@compute-02 ~]#
## Deactivate snapshot
[root@compute-02 ~]# lvchange -an -K lvmvg/lvmnova.dad2d364-3432-454d-a72d-910067e0187d
[root@compute-02 ~]# lvscan
ACTIVE '/dev/lvmvg/pool00' [<167.88 GiB] inherit
ACTIVE '/dev/lvmvg/lvmnova' [<167.88 GiB] inherit
inactive '/dev/lvmvg/lvmnova.dad2d364-3432-454d-a72d-910067e0187d' [<167.88 GiB] inherit
[root@compute-02 ~]#
|
| Code Block |
|---|
[root@controller-01 ~]# nova start dad2d364-3432-454d-a72d-910067e0187d
Request to start server dad2d364-3432-454d-a72d-910067e0187d has been accepted.
[root@controller-01 ~]# |
Cold migration for instances created on ephemeral storage
With cold migration instance will be rebooted
E.g. to migrated instance with UUID e1914906-7e33-4d30-88f0-785130fdd85d to the compute node cld-np-14:
| Code Block | ||
|---|---|---|
| ||
nova migrate e1914906-7e33-4d30-88f0-785130fdd85d --host cld-np-14.cloud.pd.infn.it |
...
| Code Block | ||
|---|---|---|
| ||
openstack server migrate --os-compute-api-version 2.30 --live --host cld-nl-19.cloud.pd.infn.it 298c94e2-e4ec-478c-8347-c77f7e1cc8df |
If source and target hypervisor aren't compatible, you will get an error message. In this case find another compatible target hypervisor or use cold migration (see above).
...
Troubleshooting
If source and target compute nodes have different settings for SELINUX (e.g. 'disabled' vs 'permissive') live migration will fail. No errors will be reported when issuing the command and no errors will be reported in the output of "openstack server event list <uuid> command". In the nova log of the compute node you will see something like:
...