Upgrade a cloud¶

Recommendations and guidelines for upgrading production clouds¶

Upgrades can be disruptive; perform them during a maintenance window
The control-plane must be upgraded before the data-plane. Simply use cou upgrade control-plane without subcommands to ensure the correct ordering of operations.
Forcing the upgrade of non-empty hypervisors will disrupt the connectivity of the VMs they are hosting; live-migrate instances away from hypervisors undergoing an upgrade if possible
Use the hypervisors subcommand to test the upgrade of a single canary machine before upgrading the rest of the data-plane.
If no issues are found after upgrading the canary machine, proceed with the upgrade.

Upgrade the whole cloud¶

To upgrade the entire OpenStack cloud, including both the control-plane and the data-plane, use:

cou upgrade

Upgrade the control-plane¶

To run an upgrade targeting only the control-plane applications use:

cou upgrade control-plane

Upgrade the data-plane¶

To run an upgrade targeting only the data-plane applications use:

cou upgrade data-plane

Note:

It’s essential to complete the upgrade of the control-plane components before being able to upgrade the data-plane.
By default, if non-empty hypervisor are identified, they are going to be excluded from the upgrade and a warning message will be shown. See the Upgrade non-empty hypervisors section for instructions on how to include them.

Upgrade the hypervisors¶

To upgrade just the hypervisors use:

# upgrade for all empty hypervisors
cou upgrade hypervisors

It’s also possible to target specific Juju availability-zones or machines:

# upgrade for hypervisors with machine ID 0 and 1 (unless they're hosting VMs)
cou upgrade hypervisors --machine "0, 1"

# upgrade for all empty hypervisors that are in zone-1
cou upgrade hypervisors --availability-zone=zone-1

Note:

Those specific filters are mutually exclusive, meaning that it’s not possible to use them together.
Since hypervisors are part of the data-plane, they won’t be upgraded unless the control-plane has already been upgraded.
By default, if non-empty hypervisor are identified, they are going to be excluded from the upgrade and a warning message will be shown. See the Upgrade non-empty hypervisors section for instructions on how to include them.

Upgrade non-empty hypervisors¶

If it’s necessary to upgrade non-empty hypervisors, use the –force option. For example:

# upgrade all data-plane applications, including hypervisors currently running instances
cou upgrade data-plane --force

# upgrade all hypervisors, even if they are hosting running instances
cou upgrade hypervisors --force

# upgrade hypervisors on machines 0 and 1, even if they are hosting running instances
cou upgrade hypervisors --machine "0, 1" --force

# upgrade all hypervisors that are in zone-1, even if they are hosting running instances
cou upgrade hypervisors --availability-zone=zone-1 --force

Note: This will disrupt connectivity for any running VM. Migrate them elsewhere before upgrading if this is undesirable.

Run interactive upgrades¶

By default, COU runs upgrade in an interactive mode, prompting the user to confirm each step.

Usage example¶

user@host:~$ cou upgrade

Full execution log: '/home/ubuntu/.local/share/cou/log/cou-20231215211917.log'Connected to 'test-model' ✔Analyzing cloud... ✔Generating upgrade plan... ✔Upgrade cloud from 'ussuri' to 'victoria' Verify that all OpenStack applications are in idle state Back up MySQL databases Archive old database data on nova-cloud-controller Control Plane principal(s) upgrade plan Upgrade plan for 'rabbitmq-server' to 'victoria' Upgrade software packages of 'rabbitmq-server' from the current APT repositories Upgrade software packages on unit 'rabbitmq-server/0' Upgrade software packages on unit 'rabbitmq-server/1' Upgrade software packages on unit 'rabbitmq-server/2' Upgrade 'rabbitmq-server' to the new channel: '3.9/stable' Change charm config of 'rabbitmq-server' 'source' to 'cloud:focal-victoria' Wait for up to 2400s for model 'test-model' to reach the idle state Verify that the workload of 'rabbitmq-server' has been upgraded ...Would you like to start the upgrade? Continue (y/N): yRunning cloud upgrade...Verify that all OpenStack applications are in idle state ✔Back up MySQL databases ✔ Upgrade plan for 'rabbitmq-server' to 'victoria' Upgrade software packages of 'rabbitmq-server' from the current APT repositories Upgrade software packages on unit 'rabbitmq-server/0' Upgrade software packages on unit 'rabbitmq-server/1' Upgrade software packages on unit 'rabbitmq-server/2' Upgrade 'rabbitmq-server' to the new channel: '3.9/stable' Change charm config of 'rabbitmq-server' 'source' to 'cloud:focal-victoria' Wait for up to 2400s for model 'test-model' to reach the idle state Verify that the workload of 'rabbitmq-server' has been upgraded Continue (y/n): yUpgrade plan for 'rabbitmq-server' to 'victoria' ✔ Upgrade plan for 'keystone' to 'victoria' Upgrade software packages of 'keystone' from the current APT repositories Upgrade software packages on unit 'keystone/0' Upgrade software packages on unit 'keystone/1' Upgrade software packages on unit 'keystone/2' Upgrade 'keystone' to the new channel: 'victoria/stable' Change charm config of 'keystone' 'openstack-origin' to 'cloud:focal-victoria' Wait for up to 2400s for model 'test-model' to reach the idle state Verify that the workload of 'keystone' has been upgraded Continue (y/n): yUpgrade software packages of 'keystone' from the current APT repositories \ ... # apply each stepUpgrade completed.

Run non-interactive upgrades¶

COU provides a non-interactive mode which suppresses user prompts and automatically continues executing each planned step. This option allows COU to be used by scripts or during upgrade testing. A quiet mode switch is also offered, which suppresses all logs and only prints important information including the generated plan and critical messages like the completion of the upgrade.

Usage examples¶

Non-interactive mode:

user@host:~$ cou upgrade --auto-approve

Full execution log: '/home/ubuntu/.local/share/cou/log/cou-20231215211717.log'Connected to 'test-model' ✔Analyzing cloud... ✔Generating upgrade plan... ✔... # the generated planRunning cloud upgrade...Verify that all OpenStack applications are in idle state ✔Back up MySQL databases ✔Upgrade software packages of 'keystone' from the current APT repositories ✔Upgrade 'keystone' to the new channel: 'victoria/stable' ✔...Upgrade completed.

Non-interactive and quiet mode:

user@host:~$ cou upgrade --auto-approve --quiet

Upgrade cloud from 'ussuri' to 'victoria' Verify that all OpenStack applications are in idle state Back up MySQL databases ...Upgrade completed.