Page History

...

Code Block

language	bash

In https://cld-config.cloud.pd.infn.it/hosts/controller-xx.cloud.pd.infn.it editare l'host sostituendo l'hostgroup "hosts_all/ComputeNodeControllerNode-Test" con "hosts_all/ComputeNodeControllerNode_Test-Test_Epoxy"

Nel controller poi eseguire
puppet agent -t

...

KEYSTONE

Code Block

language	bash

# TODO: backup database keystone

su -s /bin/sh -c "keystone-manage doctor" keystone

[root@controller-01 StartServices]# su -s /bin/sh -c "keystone-manage doctor" keystone
WARNING: `keystone.conf [cache] enabled` is not enabled.
    Caching greatly improves the performance of keystone, and it is highly
    recommended that you enable it.

su -s /bin/sh -c "keystone-manage db_sync --expand" keystone

===============================================================================================
Dopo l'aggiornamento del controller-02 e fatto ripartire httpd, si deve eseguire il comando
su -s /bin/sh -c "keystone-manage db_sync --contract" keystone

PLACEMENT

Code Block

language	bash

1) su -s /bin/sh -c "placement-manage db sync" placement 

2) accendere i servizi per keystone, placement e dashboard
systemctl start httpd.service memcached.service shibd.service

3) in cld-config modificare il file dell'HAproxy in modo che i tre servizi keystone, placement e dashboard (memcached) puntino al controller-01 commentando il controller-02 (controllare porte 5000, 5001, 443, 8778, 11211):

cp /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/servizio_httpd_acceso01_spento02.cfg /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/haproxy_el9.cfg

4) eseguire puppet sui tre haproxy 
ssh root@cld-haproxy-test-01 / 02/ 03
puppet agent -t

5) spegnere e disabilitare i servizi sul controller-02
systemctl stop httpd.service memcached.service shibd.service
systemctl disable httpd.service memcached.service shibd.service


Controllare che funzioni tutto a livello di dashboard, in particolare il calendario prenotazioni GPU (se non funziona interviene Sergio)

GLANCE

Code Block

language	bash

ATTENZIONE: controllare se c'e'un ordine per l'update di glance (si possono avere due release diverse contemporaneamente?). 
Esiste  ma e' considerato non production ready, o la doc non e' aggiornata (https://docs.openstack.org/glance/2025.1/admin/zero-downtime-db-upgrade.html) 

Per glance e' meglio non rischiare e fare il down dei servizio sui due controller quindi:

1) spegnere il servizio glance sul controller-02
systemctl stop openstack-glance-api.service
systemctl disable openstack-glance-api.service

Sul controller-01 (gia' configurato ad Epoxy perche' abbiamo girato puppet):

2) su -s /bin/sh -c "glance-manage db expand" glance

[root@controller-01 StartServices]# cat /var/log/glance/glance-manage.log 
2026-03-16 17:30:38.111 173040 INFO alembic.runtime.migration [-] Context impl MySQLImpl.
2026-03-16 17:30:38.111 173040 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.

3) su -s /bin/sh -c "glance-manage db migrate" glance

[root@controller-01 StartServices]# su -s /bin/sh -c "glance-manage db migrate" glance
2026-03-16 17:31:33.469 173073 INFO alembic.runtime.migration [-] Context impl MySQLImpl.
2026-03-16 17:31:33.470 173073 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.
Database is up to date. No migrations needed.
[root@controller-01 StartServices]# 

4) systemctl start openstack-glance-api.service

Mar 16 17:31:56 controller-01.cloud.pd.infn.it glance-api[173117]: 2026-03-16 17:31:56.058 173117 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with>


5)) Modificare l'HA proxy in modo che glance punti al controller-01
in cld-config:
cp /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/servizio_httpd_glance_acceso01_spento02.cfg /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/haproxy_el9.cfg
(controllare porta 9292)

6) eseguire puppet nei tre haproxy
puppet agent -t

=============================================================
Quanto anche il controller-02 sara' aggiornato, eseguire
su -s /bin/sh -c "glance-manage db contract" glance

...

NOVA

Code Block

language	bash

su -s /bin/sh -c "nova-status upgrade check" nova
su -s /bin/sh -c "nova-manage api_db sync" nova

in nova-manage.log
2026-03-17 10:59:31.218 208205 INFO alembic.runtime.migration [-] Context impl MySQLImpl.
2026-03-17 10:59:31.219 208205 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.


su -s /bin/sh -c "nova-manage db sync" nova

in nova-manage.log
026-03-17 11:00:31.148 208229 INFO alembic.runtime.migration [None req-22977721-3f23-4cb3-ac86-834aa11e3b59 - - - - - -] Running upgrade 13863f4e1612 -> d60bddf7a903, add_constraint_instance_share_avoid_duplicates
2026-03-17 11:00:32.539 208229 INFO alembic.runtime.migration [None req-22977721-3f23-4cb3-ac86-834aa11e3b59 - - - - - -] Running upgrade d60bddf7a903 -> 2903cd72dc14, add_tls_port_to_console_auth_tokens
2026-03-17 11:00:32.746 208229 INFO alembic.runtime.migration [None req-22977721-3f23-4cb3-ac86-834aa11e3b59 - - - - - -] Context impl MySQLImpl.
2026-03-17 11:00:32.747 208229 INFO alembic.runtime.migration [None req-22977721-3f23-4cb3-ac86-834aa11e3b59 - - - - - -] Will assume non-transactional DDL.
2026-03-17 11:00:32.755 208229 INFO alembic.runtime.migration [None req-22977721-3f23-4cb3-ac86-834aa11e3b59 - - - - - -] Running upgrade 13863f4e1612 -> d60bddf7a903, add_constraint_instance_share_avoid_duplicates
2026-03-17 11:00:33.176 208229 INFO alembic.runtime.migration [None req-22977721-3f23-4cb3-ac86-834aa11e3b59 - - - - - -] Running upgrade d60bddf7a903 -> 2903cd72dc14, add_tls_port_to_console_auth_tokens


Far partire il servizio nel controller1 
systemctl start \
    openstack-nova-api.service \
    openstack-nova-scheduler.service \
    openstack-nova-conductor.service \
    openstack-nova-novncproxy.service

Modificare l'HA proxy in modo che nova punti al controller-01
in cld-config:
cp /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/servizio_httpd_glance_nova_acceso01_spento02.cfg /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/haproxy_el9.cfg
(controllare porte 8774, 8775, 6080)

Eseguire puppet nei tre haproxy
puppet agent -t

Spegnere e disabilitare il servizio nel controller2

 systemctl stop \
    openstack-nova-api.service \
    openstack-nova-scheduler.service \
    openstack-nova-conductor.service \
    openstack-nova-novncproxy.service
 
  systemctl disable \
    openstack-nova-api.service \
    openstack-nova-scheduler.service \
    openstack-nova-conductor.service \
    openstack-nova-novncproxy.service 

==============================================================================
Quando anche il controller2 e tutti i compute saranno aggiornati, eseguire di nuovo
su -s /bin/sh -c "nova-manage db online_data_migrations" nova

NEUTRON

Code Block

language	bash

su -s /bin/sh -c "neutron-db-manage upgrade --expand" neutron

a monitor compare:
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade (expand) for neutron ...
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 0e6eff810791 -> 175fa80908e1
INFO  [alembic.runtime.migration] Running upgrade 175fa80908e1 -> 5bcb7b31ec7d
INFO  [alembic.runtime.migration] Running upgrade 5bcb7b31ec7d -> ad80a9f07c5c
  OK


Far partire il servizio 

systemctl start neutron-server.service 
systemctl start neutron-openvswitch-agent.service neutron-dhcp-agent.service \
  neutron-metadata-agent.service neutron-l3-agent.service.runtime.migration] Running upgrade 5bcb7b31ec7d -> ad80a9f07c5c
  OK


Far partire il servizio 

systemctl start neutron-server.service 
systemctl start neutron-openvswitch-agent.service neutron-dhcp-agent.service \
  neutron-metadata-agent.service neutron-l3-agent.service


N.B compare a log  2026-03-31 11:16:14.500 882189 WARNING oslo_config.cfg [-] Deprecated: Option "api_paste_config" 


#Staccare i router dal controller-02

for i in $(openstack router list -f value -c ID); do echo $i; openstack network agent list --agent-type l3 --sort-column Host --router $i --long; done 
openstack network agent remove router --l3 aa34b512-89d8-4913-aee1-9f2d2fdf124c eaa80135-6b79-44e0-b637-cef88d09b85c;

openstack network agent remove router --l3 b91764b8-58a2-4ad6-a8fc-fd20aa664571 92e8b080-f3aa-4d9f-b3d4-613e0dbfd099
openstack network agent remove router --l3 b91764b8-58a2-4ad6-a8fc-fd20aa664571 9e31c216-0635-4d21-b7aa-63fe4aee875e
openstack network agent remove router --l3 b91764b8-58a2-4ad6-a8fc-fd20aa664571 eaa80135-6b79-44e0-b637-cef88d09b85c

#verificare che l'ip sia collegato ora allo 01
ip netns exec qrouter-92e8b080-f3aa-4d9f-b3d4-613e0dbfd099 ip a | grep 172.25.27.180
ip netns exec qrouter-9e31c216-0635-4d21-b7aa-63fe4aee875e ip a | grep 90.147.77.210
ip netns exec qrouter-eaa80135-6b79-44e0-b637-cef88d09b85c ip a | grep 90.147.143.145


Modificare l'HA proxy in modo che neutron punti al controller-01
in cld-config:
cp /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/servizio_httpd_glance_nova_neutron_acceso01_spento02.cfg /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/haproxy_el9.cfg
(controllare porta 9696)

Eseguire puppet nei tre haproxy
puppet agent -t

Stoppare e disabilitare il servizio sul controller2
systemctl stop neutron-openvswitch-agent.service neutron-dhcp-agent.service \
  neutron-metadata-agent.service neutron-l3-agent.service
systemctl stop neutron-server.service

systemctl disable neutron-openvswitch-agent.service neutron-dhcp-agent.service \
  neutron-metadata-agent.service neutron-l3-agent.service
systemctl disable neutron-server.service
 

CONTROLLARE: 
[root@controller-01 neutron]# openstack server list
Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages'
Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages'
....



=========================================================================
Quando anche il controller2 sara' aggiornato eseguire il comando

su -s /bin/sh -c "neutron-db-manage upgrade --contract" neutron

CINDER

...

Code Block

language	bash
title	risultato
collapse	true

# Nell’update vengono scaricati i nuovi rpm: attenzione a questi file di configurazione

## cp /etc/openstack-dashboard/local_settings /etc/openstack-dashboard/local_settings.$REL
cp /etc/nova/nova.conf /etc/nova/nova.conf.$REL
cp /etc/placement/placement.conf /etc/placement/placement.conf.$REL
cp /etc/heat/heat.conf /etc/heat/heat.conf.$REL
cp /etc/neutron/dhcp_agent.ini /etc/neutron/dhcp_agent.ini.$REL
cp /etc/neutron/l3_agent.ini /etc/neutron/l3_agent.ini.$REL
cp /etc/neutron/metadata_agent.ini /etc/neutron/metadata_agent.ini.$REL
cp /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugins/ml2/ml2_conf.ini.$REL
cp /etc/neutron/plugins/ml2/openvswitch_agent.ini /etc/neutron/plugins/ml2/openvswitch_agent.ini.$REL
cp /etc/keystone/keystone.conf /etc/keystone/keystone.conf.$REL
cp /etc/glance/glance-api.conf /etc/glance/glance-api.conf.$REL
cp /etc/cinder/cinder.conf /etc/cinder/cinder.conf.$REL
cp /etc/httpd/conf.d/auth_openidc.conf /etc/httpd/conf.d/auth_openidc.conf.$REL

## mv -f /etc/openstack-dashboard/local_settings.rpmnew /etc/openstack-dashboard/local_settings
mv -f /etc/nova/nova.conf.rpmnew /etc/nova/nova.conf
mv -f /etc/placement/placement.conf.rpmnew /etc/placement/placement.conf
mv -f /etc/heat/heat.conf.rpmnew /etc/heat/heat.conf
mv -f /etc/neutron/dhcp_agent.ini.rpmnew /etc/neutron/dhcp_agent.ini
mv -f /etc/neutron/l3_agent.ini.rpmnew /etc/neutron/l3_agent.ini
mv -f /etc/neutron/metadata_agent.ini.rpmnew /etc/neutron/metadata_agent.ini
mv -f /etc/neutron/plugins/ml2/ml2_conf.ini.rpmnew /etc/neutron/plugins/ml2/ml2_conf.ini
mv -f /etc/neutron/plugins/ml2/openvswitch_agent.ini.rpmnew /etc/neutron/plugins/ml2/openvswitch_agent.ini
mv -f /etc/keystone/keystone.conf.rpmnew /etc/keystone/keystone.conf
mv -f /etc/glance/glance-api.conf.rpmnew /etc/glance/glance-api.conf
mv -f /etc/cinder/cinder.conf.rpmnew /etc/cinder/cinder.conf
mv -f /etc/httpd/conf.d/auth_openidc.conf.rpmnew /etc/httpd/conf.d/auth_openidc.conf

Cambiare classe in Foreman con Epoxy

Code Block

language	bash

Da pagina web di Foreman, modificare la classe puppet del controller selezionando Epoxy
in https://cld-config.cloud.pd.infn.it/hosts/controller-xx.cloud.pd.infn.it editare l'host sostituendo l'hostgroup "hosts_all/ComputeNodeControllerNode-Test" con "hosts_all/ComputeNodeControllerNode_Test-Test_Epoxy"

Nel controller poi eseguire
puppet agent -t 

A questo punto tutti i servizi sono configurati

girare puppet nel nodo
Code Block
language shell
puppet agent -t

...

attivare i servizi modificando service.pp per far partire i servizi

Code Block

language	shell

# modificare in service.pp tutti i servizi
   ensure      => running,
   enable      => true,
# e committare in git

riabilitare puppet nel nodo
Code Block
language shell
systemctl start puppet systemctl enable puppet

modificare il cld-config il file di haproxy per utilizzare i due controller

Code Block

language	shell

cp  /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/haproxy_el9.cfg.orig  /etc/puppetlabs/code/environments/production/modules/cloudtest_haproxy/files/haproxy_el9.cfg

eseguire puppet nei tre haproxy

fare i contract o online-migration del db per i servizi che lo richiedono

Code Block

language	shell

# Dopo l'aggiornamento del controller-02
su -s /bin/sh -c "keystone-manage db_sync --contract" keystone    

su -s /bin/sh -c "glance-manage db contract" glance

su -s /bin/sh -c "neutron-db-manage upgrade --contract" neutron

su -s /bin/sh -c "cinder-manage db online_data_migrations" cinder

    
-->> QUI  31/03 <--

verificare la creazione di nuove VM. Se la contastualizzazione non funziona dando errore di connessione al metadata server allora controllare se compare l'agente tra i network agent list e vedere quando l'heartbeat e' stato eseguito l'ultima volta.

se la data e' vecchia, rimuovere l'agent dai due controller e fare il reboot

Code Block

language	shell

[root@controller-02 nova]# openstack network agent list
Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages'
Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages'
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                             | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+-------+---------------------------+
| 03b6f400-d961-42cd-9df9-89e87dd58ca9 | Open vSwitch agent | controller-02.cloud.pd.infn.it   | None              | :-)   | UP    | neutron-openvswitch-agent |
| 10f518b3-d9a6-4adf-a482-20723682b5f5 | Metadata agent     | controller-02.cloud.pd.infn.it   | None              | XXX   | UP    | neutron-metadata-agent    |
| 3241aa58-f697-478c-bacc-4e10d7cc43e7 | Open vSwitch agent | controller-01.cloud.pd.infn.it   | None              | XXX   | UP    | neutron-openvswitch-agent |
| 7b34d1ad-99a7-4ca8-a1e6-82a90737a635 | Open vSwitch agent | t2-cld-nat-test.cloud.pd.infn.it | None              | :-)   | UP    | neutron-openvswitch-agent |
| 7c026284-8b62-420d-9163-464c3b28bf24 | Open vSwitch agent | compute-01.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
| 940d868e-8605-42e5-a731-b07e2a2a311e | DHCP agent         | controller-01.cloud.pd.infn.it   | nova              | XXX   | UP    | neutron-dhcp-agent        |
| aa34b512-89d8-4913-aee1-9f2d2fdf124c | L3 agent           | controller-01.cloud.pd.infn.it   | nova              | XXX   | UP    | neutron-l3-agent          |
| b60f9a09-06ad-4562-b1c9-72ef265200a6 | DHCP agent         | controller-02.cloud.pd.infn.it   | nova              | :-)   | UP    | neutron-dhcp-agent        |
| b91764b8-58a2-4ad6-a8fc-fd20aa664571 | L3 agent           | controller-02.cloud.pd.infn.it   | nova              | :-)   | UP    | neutron-l3-agent          |
| be79d4c8-f24d-47f9-876b-09ed34614dc2 | Open vSwitch agent | compute-03.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
| df3074d3-0add-4f78-a5f4-fde900e764f2 | Open vSwitch agent | compute-02.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
| fd8b02e9-ca5f-43d4-b1fc-31163ba2b7b3 | Open vSwitch agent | compute-04.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+-------+---------------------------+

[root@controller-02 nova]# openstack network agent show 10f518b3-d9a6-4adf-a482-20723682b5f5
Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages'
Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages'
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                 |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up    | UP                                                                                                                                                    |
| agent_type        | Metadata agent                                                                                                                                        |
| alive             | XXX                                                                                                                                                   |
| availability_zone | None                                                                                                                                                  |
| binary            | neutron-metadata-agent                                                                                                                                |
| configuration     | {'log_agent_heartbeats': False, 'metadata_proxy_socket': '/var/lib/neutron/metadata_proxy', 'nova_metadata_host': '192.168.60.24',                    |
|                   | 'nova_metadata_port': 8775}                                                                                                                           |
| created_at        | 2018-11-06 09:30:53                                                                                                                                   |
| description       | None                                                                                                                                                  |
| ha_state          | None                                                                                                                                                  |
| host              | controller-02.cloud.pd.infn.it                                                                                                                        |
| id                | 10f518b3-d9a6-4adf-a482-20723682b5f5                                                                                                                  |
| last_heartbeat_at | 2026-03-17 10:41:41                                                                                                                                   |
| resources_synced  | None                                                                                                                                                  |
| started_at        | 2026-03-09 10:56:21                                                                                                                                   |
| topic             | N/A                                                                                                                                                   |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@controller-02 nova]#

Dopo aver eliminato i metadata agent e fatto il reboot la situazione e' la seguente:

Code Block

language	shell

[root@controller-01 ~]# openstack network agent list
Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages'
Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages'
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                             | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+-------+---------------------------+
| 03b6f400-d961-42cd-9df9-89e87dd58ca9 | Open vSwitch agent | controller-02.cloud.pd.infn.it   | None              | :-)   | UP    | neutron-openvswitch-agent |
| 3241aa58-f697-478c-bacc-4e10d7cc43e7 | Open vSwitch agent | controller-01.cloud.pd.infn.it   | None              | :-)   | UP    | neutron-openvswitch-agent |
| 7b34d1ad-99a7-4ca8-a1e6-82a90737a635 | Open vSwitch agent | t2-cld-nat-test.cloud.pd.infn.it | None              | :-)   | UP    | neutron-openvswitch-agent |
| 7c026284-8b62-420d-9163-464c3b28bf24 | Open vSwitch agent | compute-01.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
| 940d868e-8605-42e5-a731-b07e2a2a311e | DHCP agent         | controller-01.cloud.pd.infn.it   | nova              | :-)   | UP    | neutron-dhcp-agent        |
| aa34b512-89d8-4913-aee1-9f2d2fdf124c | L3 agent           | controller-01.cloud.pd.infn.it   | nova              | :-)   | UP    | neutron-l3-agent          |
| b60f9a09-06ad-4562-b1c9-72ef265200a6 | DHCP agent         | controller-02.cloud.pd.infn.it   | nova              | :-)   | UP    | neutron-dhcp-agent        |
| b91764b8-58a2-4ad6-a8fc-fd20aa664571 | L3 agent           | controller-02.cloud.pd.infn.it   | nova              | :-)   | UP    | neutron-l3-agent          |
| be79d4c8-f24d-47f9-876b-09ed34614dc2 | Open vSwitch agent | compute-03.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
| df3074d3-0add-4f78-a5f4-fde900e764f2 | Open vSwitch agent | compute-02.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
| fd8b02e9-ca5f-43d4-b1fc-31163ba2b7b3 | Open vSwitch agent | compute-04.cloud.pd.infn.it      | None              | :-)   | UP    | neutron-openvswitch-agent |
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+-------+---------------------------+

...

Mettere in drain un nodo alla volta.

openstack compute service set --disable compute-01.cloud.pd.infn.it nova-compute

openstack compute service list

~~Per il singolo nodo in drain, migrare le VM con live migration quando possibile (altrimenti si spegne e si migra)~~

In foreman cambiamo la classe per Epoxy

...

Space shortcuts

Page tree

Versions Compared

Old Version 59

New Version Current

Key