Questa guida spiega la procedura per aggiornare i 2 controller da centos8 stream a AlmLinux9
Backup dei 2 controller
Su cld-log:
rbd create --size 2T --data-pool misc metadata-misc/backup-cld-ctrl-disk rbd map metadata-misc/backup-cld-ctrl-disk mkfs.xfs /dev/rbd1 mkdir /backup-cld-ctrl mount /dev/rbd1 /backup-cld-ctrl mkdir /backup-cld-ctrl/cld-ctrl-01 mkdir /backup-cld-ctrl/cld-ctrl-02 rsync -axHAv --progress cld-ctrl-01:/ /backup-cld-ctrl/cld-ctrl-01 rsync -axHAv --progress cld-ctrl-02:/ /backup-cld-ctrl/cld-ctrl-02
Fatto anche backup di /root, /etc, /usr/local
Messi i tarball in cld-config:/BACKUP
Salvato anche l'output di "nmcli", "nmcli dev" "nmcli conn"
"Disabilitazione" di cld-ctrl-02
Modifico la configurazione di haproxy in modo da usare un solo controller (cld-ctrl-01)
Si fa via puppet. Si puo' usare il file haproxy.cfg.solo-01-2024-Aug-28
In foreman metto cld-ctrl-02 in hosts_all
Spengo e disabilito i servizi su cld-ctrl-02:
[root@cld-ctrl-02 ~]# systemctl stop puppet [root@cld-ctrl-02 ~]# systemctl disable puppet [root@cld-ctrl-02 ~]# cd StartServices # . all.sh stop # . all.sh disable
Clone di cld-ctrl-02
Fatto da Sergio Traldi
Reinstallazione cld-ctrl-02
Reinstallo cld-ctrl-02 con AlmaLinux9
Usato come parition table: kickstart default - swap 32GB
Configurazione rete sui cld-ctrl-02
nmcli con mod ens1f0np0 ipv4.method manual ipv4.addr "192.168.61.106/24" nmcli con mod ens1f0np0 802-3-ethernet.mtu 9000 nmcli con up ens1f0np0 nmcli con mod ens1f0np0 connection.autoconnect yes #nmcli con add type vlan ifname ens1f0np0.40 dev ens1f1np1 id 40 #nmcli con mod vlan-ens1f1np1.303 ens1f0np0.40 disabled #nmcli con mod vlan-ens1f1np1.888 ens1f0np0.40 disabled nmcli con add type vlan ifname ens1f1np1.303 dev ens1f1np1 id 303 nmcli con add type vlan ifname ens1f1np1.401 dev ens1f1np1 id 401 nmcli con add type vlan ifname ens1f1np1.888 dev ens1f1np1 id 888 nmcli con mod vlan-ens1f1np1.303 ipv4.method disabled nmcli con mod vlan-ens1f1np1.401 ipv4.method disabled nmcli con mod vlan-ens1f1np1.402 ipv4.method disabled nmcli con mod vlan-ens1f1np1.888 ipv4.method disabled nmcli con mod vlan-ens1f1np1.303 ipv6.method disabled nmcli con mod vlan-ens1f1np1.401 ipv6.method disabled nmcli con mod vlan-ens1f1np1.402 ipv6.method disabled nmcli con mod vlan-ens1f1np1.888 ipv6.method disabled ip link set ens1f1np1 mtu 9000 #ip link set ens1f0np0 mtu 9000 reboot
Setting preliminari su cld-ctrl-02
Aumento dei max file descriptors
cat << EOF >> /etc/security/limits.conf * soft nofile 16384 * hard nofile 16384 EOF
Modifiche a sysctl.conf
cat << EOF >> /etc/sysctl.conf net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 EOF sysctl -p
Disabilitazione GRO
Per ogni connessione associata a un bridge di rete esterna:
nmcli con modify vlan-ens1f1np1.303 ethtool.feature-gro off nmcli con modify vlan-ens1f1np1.401 ethtool.feature-gro off nmcli con modify vlan-ens1f1np1.402 ethtool.feature-gro off nmcli con modify vlan-ens1f1np1.888 ethtool.feature-gro off
Controllare il setting gro per ogni interfaccia, che deve essere off
# nmcli con show vlan-ens1f1np1.303 | grep gro ethtool.feature-gro: off
Installazione pacchetti su cld-ctrl-02
Warning: installare openmanage solo dopo avere runnato puppet (altrimenti ci sono problemi di dipendenze con libxerces-c)
Mi assicuro che epel sia disabilitato
Poi:
dnf config-manager --set-enabled crb yum install centos-release-openstack-yoga yum clean all yum update yum install python3-openstackclient yum install openstack-selinux yum install memcached python3-memcached yum install openstack-keystone httpd python3-mod_wsgi yum install openstack-glance yum install openstack-placement-api yum install openstack-nova-api openstack-nova-conductor openstack-nova-novncproxy openstack-nova-scheduler yum install openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch ebtables yum install openstack-dashboard yum install openstack-cinder yum install openstack-ec2-api yum install openstack-heat-api openstack-heat-api-cfn openstack-heat-engine yum install python3-mysqlclient --enablerepo=epel yum install mod_ssl yum install mod_auth_openidc
Altre operazioni da fare su cld-ctrl-02
Memcached
Modificare la riga options di /etc/sysconfig/memcached mettendo l'IP della rete di mgmt
[root@cld-ctrl-02 ~]# grep -i opt /etc/sysconfig/memcached OPTIONS="-l 192.168.60.106" #OPTIONS="-l 127.0.0.1,::1" [root@cld-ctrl-02 ~]#
Glance
Glance (non sono sicurissimo che questa cosa serva ancora):
mkdir /Images mkdir /Images/staging chown glance.glance /Images/ chown glance.glance /Images/staging
Firewall
Editare /etc/firewalld/firewalld.conf settando:
FirewallBackend=iptables
systemctl restart firewalld
Poi runnare lo script firewall.sh
CA e certificati
wget https://repository.egi.eu/sw/production/cas/1/current/repo-files/egi-trustanchors.repo -O /etc/yum.repos.d/EGI-trustanchors.repo yum install ca-policy-egi-core yum install fetch-crl --enablerepo=epel
TBC: start di fetch-ctrl
Installare i file (copiandoli da controller-01):
/etc/grid-security/keystone-*.pem
/etc/grid-security/horizon*.pem
cd /etc/grid-security ln -s horizon-infn-cert.pem hostcert.pem ln -s horizon-infn-key.pem hostkey.pem #ln -s horizon-infn-cert.pem cert.pem #ln -s horizon-infn-key.pem key.pem
Keystone
mkdir -p /etc/keystone/fernet-keys && chown keystone.keystone /etc/keystone/fernet-keys && chmod 700 /etc/keystone/fernet-keys mkdir -p /etc/keystone/credential-keys && chown keystone.keystone /etc/keystone/credential-keys && chmod 700 /etc/keystone/credential-keys scp controller-01:/etc/keystone/fernet-keys/* /etc/keystone/fernet-keys scp controller-01:/etc/keystone/credential-keys/* /etc/keystone/credential-keys chown keystone.keystone /etc/keystone/fernet-keys/* chown keystone.keystone /etc/keystone/credential-keys/*
Openvswitch
systemctl start openvswitch ovs-vsctl add-br br-ex ovs-vsctl add-br br-ex-2 ovs-vsctl add-br br-ex-3 ovs-vsctl add-br br-ex-4 ovs-vsctl add-port br-ex ens1f1np1.303 ovs-vsctl add-port br-ex-2 ens1f1np1.888 ovs-vsctl add-port br-ex-3 ens1f1np1.401 ovs-vsctl add-port br-ex-4 ens1f1np1.402
Configurazione via puppet su cld-ctrl-02:
Spengo puppet
systemctl stop puppet
In foreman sposto cld-ctrl-02 sull'hostgroup hosts_all/ControllerNode-Prod_Yoga
e forzo l'esecuzione di puppet
puppet agent -t
Poi:
systemctl start puppet systemctl enable puppet
Operazioni finali su cld-ctrl-02
- Reinstallazione openmanage
Warning: installare openmanage solo dopo avere runnato puppet (altrimenti ci sono problemi di dipendenze con libxerces-c)
- Cron check cert. Copio da controller-01:
- /etc/cron.d/check_certificates
- /usr/local/bin/check_certs.sh
- /usr/local/bin/check_ssl_cert (prenderlo da https://raw.githubusercontent.com/matteocorti/check_ssl_cert/master/check_ssl_cert o da un altro nodo aggiornato: quello che c'era prima non va bene)
- Calendario GPUs
- Aggiungere la chiave di nagios@cld-nagios in authorized-keys di root e verificare che l'ssh funzioni senza password
- Ripristinare /root/StartServices
- Ripristinare /root/admin-openrc.sh
"Riabilitazione" di cld-ctrl-02
Modifico la configurazione di haproxy in modo da usare entrambi i controller
Si fa via puppet.
"Disabilitazione" di controller-01
Modifico la configurazione di haproxy in modo da usare un solo controller (cld-ctrl-02)
Si fa via puppet. Si possono usare i file haproxy.cfg.only02-2024-07-15 (per cld-haproxy-test-01 e 02) e haproxy_el9.cfg.only02-2024-07-15 (per cld-haproxy-test-03)
In foreman metto controller-01 in hosts_all
Spengo e disabilito i servizi su controller-01:
[root@controller-01 ~]# systemctl stop puppet [root@controller-01 ~]# systemctl disable puppet [root@controller-01 ~]# cd StartStopServices # . complete.sh stop # . complete.sh disable
Reinstallazione controller-01
Reinstallo controller-02 con AlmaLinux9
Usato come parition table: kickstart default - swap 32GB
Configurazione rete sui controller-01
nmcli con mod eno3 ipv4.method manual ipv4.addr "192.168.61.42/24" nmcli con mod eno3 802-3-ethernet.mtu 9000 nmcli con up eno3 nmcli con mod eno3 connection.autoconnect yes nmcli con add type vlan ifname eno3.303 dev eno3 id 303 nmcli con add type vlan ifname eno3.401 dev eno3 id 401 nmcli con add type vlan ifname eno3.888 dev eno3 id 888 nmcli con mod vlan-eno3.303 ipv4.method disabled nmcli con mod vlan-eno3.401 ipv4.method disabled nmcli con mod vlan-eno3.888 ipv4.method disabled nmcli con mod vlan-eno3.303 ipv6.method disabled nmcli con mod vlan-eno3.401 ipv6.method disabled nmcli con mod vlan-eno3.888 ipv6.method disabled ip link set eno3 mtu 9000 reboot
Setting preliminari su controller-01
Aumento dei max file descriptors
cat << EOF >> /etc/security/limits.conf * soft nofile 16384 * hard nofile 16384 EOF
Modifiche a sysctl.conf
cat << EOF >> /etc/sysctl.conf net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 EOF sysctl -p
Disabilitazione GRO
Per ogni connessione associata a un bridge di rete esterna:
nmcli con modify vlan-eno3.303 ethtool.feature-gro off nmcli con modify vlan-eno3.401 ethtool.feature-gro off nmcli con modify vlan-eno3.888 ethtool.feature-gro off
Controllare il setting gro per ogni interfaccia, che deve essere off
# nmcli con show vlan-eno3.303 | grep gro ethtool.feature-gro: off
Installazione pacchetti su controller-01
Warning: installare openmanage solo dopo avere runnato puppet (altrimenti ci sono problemi di dipendenze con libxerces-c)
Mi assicuro che epel sia disabilitato
Poi:
dnf config-manager --set-enabled crb yum install centos-release-openstack-yoga yum clean all yum update yum install python3-openstackclient yum install openstack-selinux yum install memcached python3-memcached yum install openstack-keystone httpd python3-mod_wsgi yum install openstack-glance yum install openstack-placement-api yum install openstack-nova-api openstack-nova-conductor openstack-nova-novncproxy openstack-nova-scheduler yum install openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch ebtables yum install openstack-dashboard yum install openstack-cinder yum install openstack-ec2-api yum install openstack-heat-api openstack-heat-api-cfn openstack-heat-engine yum install python3-mysqlclient --enablerepo=epel yum install mod_ssl yum install mod_auth_openidc
Altre operazioni da fare su controller-01
Memcached
Modificare la riga options di /etc/sysconfig/memcached mettendo l'IP della rete di mgmt
[root@controller-01 ~]# grep -i opt /etc/sysconfig/memcached OPTIONS="-l 192.168.60.42" #OPTIONS="-l 127.0.0.1,::1" [root@controller-02 ~]#
Glance
Glance (non sono sicurissimo che questa cosa serva ancora):
mkdir /Images mkdir /Images/staging chown glance.glance /Images/ chown glance.glance /Images/staging
Firewall
Editare /etc/firewalld/firewalld.conf settando:
FirewallBackend=iptables
systemctl restart firewalld
Poi runnare lo script firewall.sh
CA e certificati
wget https://repository.egi.eu/sw/production/cas/1/current/repo-files/egi-trustanchors.repo -O /etc/yum.repos.d/EGI-trustanchors.repo yum install ca-policy-egi-core yum install fetch-crl --enablerepo=epel
TBC: start di fetch-ctrl
Installare i file (copiandoli da controller-01):
/etc/grid-security/hostcert.pem e /etc/grid-security/hostkey.pem
/etc/grid-security/keystone-*.pem
/etc/grid-security/horizon*.pem
ln -s /etc/grid-security/horizon-unipd-cert.pem /etc/grid-security/unipd-cert.pem ln -s /etc/grid-security/horizon-unipd-key.pem /etc/grid-security//unipd-key.pem
Keystone
mkdir -p /etc/keystone/fernet-keys && chown keystone.keystone /etc/keystone/fernet-keys && chmod 700 /etc/keystone/fernet-keys mkdir -p /etc/keystone/credential-keys && chown keystone.keystone /etc/keystone/credential-keys && chmod 700 /etc/keystone/credential-keys scp cld-ctrl-02:/etc/keystone/fernet-keys/* /etc/keystone/fernet-keys scp cld-ctrl-02:/etc/keystone/credential-keys/* /etc/keystone/credential-keys chown keystone.keystone /etc/keystone/fernet-keys/* chown keystone.keystone /etc/keystone/credential-keys/*
Openvswitch
systemctl start openvswitch ovs-vsctl add-br br-ex ovs-vsctl add-br br-ex-2 ovs-vsctl add-br br-ex-3 ovs-vsctl add-port br-ex eno3.303 ovs-vsctl add-port br-ex-2 eno3.888 ovs-vsctl add-port br-ex-3 eno3.401
Configurazione via puppet su controller-01:
Spengo puppet
systemctl stop puppet
In foreman sposto controller-01 sull'hostgroup hosts_all/ControllerNode-Test_Yoga
e forzo l'esecuzione di puppet
puppet agent -t
Poi:
systemctl start puppet systemctl enable puppet
Operazioni finali su controller-01
- Reinstallazione openmanage
Warning: installare openmanage solo dopo avere runnato puppet (altrimenti ci sono problemi di dipendenze con libxerces-c)
- Cron check cert. Copio da cld-ctrl-02:
- /etc/cron.d/check_certificates
- /usr/local/bin/check_certs.sh
- /usr/local/bin/check_ssl_cert (prenderlo da https://raw.githubusercontent.com/matteocorti/check_ssl_cert/master/check_ssl_cert o da un altro nodo aggiornato: quello che c'era prima non va bene)
- Calendario GPUs
- Aggiungere la chiave di nagios@cld-nagios in authorized-keys di root e verificare che l'ssh funzioni senza password
- Ripristinare /root/StartServices
- Ripristinare /root/admin-openrc.sh
Riabilitazione di entrambi i controller in HAproxy
Cambio la conf di HAproxy in modo da abilitare entrambi i controller