View Source

This guide explains how to reinstall a compute node from Centos8 Stream-Yoga to AlmaLinux9-Yoga

In the following instructions the node cld-np-19, an INFN node, is the one to be reinstallated

First of all disable the compute node, so that no new VMs will be instantiated on this compute node:

openstack compute service set --disable cld-np-19.cloud.pd.infn.it nova-compute

To check if the compute node was actually disabled:

openstack compute service list

Then finds all the virtual machines instantiated on this compute node:

openstack server list --all --host cld-np-19.cloud.pd.infn.it

Then migrate each VM instantiated on this compute node to other hypervisors.

To avoid the migration of the same virtual machine multiple times, please migrate the VM on a compute node that was already migrated to AlmaLinux9

The migration of a virtual machine usually requires a downtime (~ 15') of that VM, unless the VM was created using a volume. So before migrating the VM you must agree this operation with the relevant owner.

E.g. to find the owner of a VM with UUID be832766-7996-4ccf-a9c1-5eec2e5f8fd3:

[root@cld-ctrl-01 ~]# openstack server show be832766-7996-4ccf-a9c1-5eec2e5f8fd3 | grep user_id
| user_id                             | 7681252c430040e7bb96c7c4f7c88464                            |
[root@cld-ctrl-01 ~]# openstack user show 7681252c430040e7bb96c7c4f7c88464
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| email               | Ysabella.Ong@lnl.infn.it         |
| enabled             | True                             |
| id                  | 7681252c430040e7bb96c7c4f7c88464 |
| name                | ysaong@infn.it                   |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

To migrate a virtual machine to another compute node, there are two possible procedures:

Procedure to be used if the VM was created using a volume (the migration procedure doesn't require a downtime)
Procedure to be used if the VM was created using ephemeral storage (the migration procedure requires a downtime)

Install the operating system using foreman. For this use the hosts_all foreman hostgroup
Disable SELinux
Configure the management and data network
Stop puppet:

systemctl stop puppet

In foreman move the host under the ComputeNode-Prod_Yoga hostgroup
Run puppet manually:
puppet agent -t
If the configuration fails reporting

error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key  
error: "net.bridge.bridge-nf-call-iptables" is an unknown key  
error: "net.bridge.bridge-nf-call-arptables" is an unknown key

then please issue:

modprobe br_netfilter

and then rerun puppet

In the configuration fails reporting a problem because of a wrong dependency required by swtpm, then issue:

mv /etc/yum.repos.d/advanced-virtualization.repo /etc/yum.repos.d/advanced-virtualization.repo.old
yum install centos-release-advanced-virtualization

and then rerun puppet

Disable the compute node so that it doesn't accept new VMs:

openstack compute service set --disable cld-np-19.cloud.pd.infn.it nova-compute

Start and enablepuppet:

systemctl start puppet; systemctl enable puppet

Enable the checks in Nagios for this host
Wait till all checks are ok (in particular the VM network and volume one: remember to enable the node just for the time to force the run from Nagios; othwerwise the check will fail.)
Add this node in the following scripts on cld-ctrl-01: /usr/local/bin/display_usage_of_hypervisors.sh, /usr/local/bin/host_aggregate.sh, /usr/local/bin/new_project.sh, /usr/local/bin/free_resources_compute_nodes.sh
Add this node in the /etc/cron.daily/vm-log.sh script on cld-log
Create the directory /var/disk-iscsi/qemu/cld-np-19 in cld-log
Verify that a ssh from root@cld-log to root@cld-np-19 works without requiring password
Stop puppet on the 2 controller nodes:
systemctl stop puppet
Add the new compute node in cld-config:/var/puppet/puppet_yoga/controller_yoga/templates/aai_settings.py.erb
Run puppet on the first controller node (this will trigger a restart of httpd):
puppet agent -t
Run puppet on the second controller node (this will trigger a restart of httpd):
puppet agent -t
Start puppet on the two controller nodes:
systemctl start puppet

Enable the host:

openstack compute service set --enable cld-np-19.cloud.pd.infn.it nova-compute

Add the host to the 'admin' aggregate

Run the aggregate_manage.sh script to add the node in the relevant hostgroups, e.g.:
/usr/local/bin/aggregate_manage.sh add cld-np-19.cloud.pd.infn.it INFN

Related articles