# Upgrade Management Plane and Hosts
The upgrade of the self-hosted space.vars.product\_name is a two-phase, sequential process:
* **Management Plane:** Upgrading the Management Plane updates core services, management APIs, region-specific configurations, and orchestration components.
* **Hosts**: Once the management plane has been upgraded, hosts in each region are being upgraded to ensure compatibility, seamless communication, and completion of the overall upgrade process
Before you begin the upgrade, ensure you meet all prerequisites.
## Prerequisites for Upgrade
Before you upgrade, complete the following checks to ensure your environment is ready.
### Verify Management Plane Health
Before you upgrade, verify that the Management Plane is healthy. Run `airctl status` and confirm that `region health` shows as `Ready`. If any regions are not ready, resolve all issues before continuing. For example, pods may be in a `CrashLoopBackOff` state.
### Manual Backup
You must perform a [Manual Backup Procedure](https://docs.platform9.com/private-cloud-director/getting-started/backup-and-restore#manual-backup-procedure) of your management plane infrastructure before starting the upgrade. This backup is critical for recovery.
## Upgrade Management Plane and Cluster
Upgrade your space.vars.product\_name environment to access the latest features, security patches, and performance improvements through this two-step process.
### Upgrade Management Plane
Updating the Management Plane running on the Management Cluster ensures multiple components are automatically upgraded as part of an `airctl upgrade` process.
Here are the components that will be affected.
* Core services and applications.
* Management APIs and user interfaces.
* Region-specific configurations and services.
* Service coordination and orchestration components.
#### The Impact of Management Plane Upgrade
{% hint style="info" %}
**NOTE**
Temporary service interruptions may occur during the upgrade process.
{% endhint %}
* The user interface will be unavailable during the upgrade.
* All API services will be temporarily unavailable.
* Create, Read, Update, and Delete operations will not be possible during this time.
* **Running workloads on VMs will not be impacted.**
#### Step 1: Download and run the Installer Artifacts
`airctl` is the command-line installer for space.vars.product\_name. Run the following commands **only on a verified management cluster node** to download `airctl` with the required installation artifacts.
{% hint style="info" %}
**Air-gapped environments only**
If the management node does not have internet access, download the installer artifacts on a jump host with internet connectivity and transfer them to the management node.
Follow the artifact download steps from the air-gapped installation guide, then skip this step and continue using the copied artifacts.
{% endhint %}
1. Download the Installer Script and Artifacts using the following command.
```bash
curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/latest/index.txt | awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/latest/" $NF "\" -o ${HOME}/" $NF}' | bash
```
You can choose to download a specific version of the installer script and artifact.
For example, you can replace `latest` with a specific version, such as `v-2026.1.1-4444155`
Here is an example of a modified command:
{% tabs %}
{% tab title="Example" %}
```bash
curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/v-2026.1.1-4444155/index.txt | \
awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/v-2026.1.1-4444155/" $NF "\" -o ${HOME}/" $NF}' | bash
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**NOTE**
Replace `` with your user key.
{% endhint %}
2. Make the Installer Executable using the following command.
```bash
chmod +x ./install-pcd.sh
```
3. Run the Installation Script using the following command.
```bash
./install-pcd.sh `cat version.txt`
```
The command installs the specific version of `airctl` based on the `version.txt` file.
4. Create symbolic links for `airctl` and its configuration file:
```bash
sudo rm -f /usr/bin/airctl # delete any existing file or symlink
sudo ln -s /opt/pf9/airctl/airctl /usr/bin/airctl
ln -s /opt/pf9/airctl/conf/airctl-config.yaml $HOME/airctl-config.yaml
```
#### Proxy Configuration (Optional)
If your environment uses a network proxy, update the values `/opt/pf9/airctl/conf/helm_values/bork.template.yml` again, as the changes would be lost after a new build is installed. Update it again as shown below:
```bash
cat /opt/pf9/airctl/conf/helm_values/kplane.template.yml | grep proxy
# Sample output:
https_proxy: "http://squid.platform9.horse:3128"
http_proxy: "http://squid.platform9.horse:3128"
no_proxy: "10.149.106.11,10.149.106.12,10.149.106.13,10.149.106.14,10.149.106.15,10.149.106.16,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.platform9.localnet,.default.svc"
```
The `no_proxy` list should include the `master-ips`, `worker-ips`, `external-ip4`, `master-vip4`, and any other addresses where traffic should not be routed through a proxy server.
{% hint style="info" %}
**NOTE**
The upgrade-cluster step is skipped in this release, as the Kubernetes version remains unchanged at **v1.30**.
{% endhint %}
{% hint style="info" %}
**Air-gapped environments only**
Ensure that all required container images for the target version are already available in the private registry accessible by the cluster nodes before running the upgrade.
If not already done, download the image list for the target version and push the images to your private registry. Refer to the air-gapped installation guide for detailed steps.
{% endhint %}
#### Step 2: Upgrade all regions
To upgrade **all** regions set up on your space.vars.product\_name, execute the following command.
```bash
airctl upgrade
```
**Optionally**, you can upgrade a specific region by replacing `` with your target region.
```bash
airctl upgrade --region
```
Here is the modified example command.
{% tabs %}
{% tab title="Example" %}
```json
# airctl upgrade --region Region1
INFO rollback state directory /tmp/airctl_ddu_backup_Region1_240020739
INFO Saving the helm revisions to the state file
INFO --- backing up region--- Region1
INFO Archive created successfully for Region1. Backup backup.tar.gz saved to /tmp/airctl_ddu_backup_Region1_240020739
INFO --- moving old state file to /tmp/airctl_ddu_backup_Region1_240020739/state.yaml ---
INFO --- moving old kplane_values.yaml file to /tmp/airctl_ddu_backup_Region1_240020739/kplane_values.yaml ---
SUCCESS Upgrading region Region1
upgrade done
```
{% endtab %}
{% endtabs %}
To monitor and diagnose the upgrade logs, add `--verbose`.
Here is the sample command.
```bash
airctl upgrade --verbose
```
#### Verify: Upgrade Success on Management Plane
After completing the upgrade for the management plane, verify that you can continue to access the user interface, and then verify the deployment status across all regions.
Run the following command:
```bash
airctl status
```
Here is the sample output.
{% tabs %}
{% tab title="Example" %}
```bash
------------- deployment details ---------------
fqdn: airctl-1-4206457-802.platform9.localnet
region: airctl-1-4206457-802
deployment status: ready
region health: ✅ Ready
version: PCD v-2026.1.1-4444155
-------- region service status ----------
desired services: 30
ready services: 30
------------- deployment details ---------------
fqdn: airctl-1-4206457-802-mel.platform9.localnet
region: airctl-1-4206457-802-mel
deployment status: ready
region health: ✅ Ready
version: PCD v-2026.1.1-4444155
-------- region service status ----------
desired services: 84
ready services: 84
```
{% endtab %}
{% endtabs %}
After a successful upgrade and verification, space.vars.product\_name it **does not support rolling back** to a previous version.
## Upgrade the Hosts
After the management plane is successfully upgraded, the hosts running in each of your regions must be updated. The agents running on the Host manage communication between hosts and the management plane.
### The Impact from Hosts Upgrade
* No `servHostavailability` is expected during the host upgrade.
* Workloads running on VMs will continue to operate without disruption.
#### Step 1: Record the Current Version of Packages on the Host
Before you begin the upgrade, execute the following command to record the current package versions:
{% tabs %}
{% tab title="Example" %}
```bash
$ airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml
Getting host statuses...
Getting host statuses for region: [REGION_NAME]
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| Host Name | IP Addresses | Host ID | Host Agent Version | Status | Agent Status | Apps |
| [HOSTNAME1] | 172.29.32.34, 192.168.122.1, 10.0.11.10 | [HOST1_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-cindervolume-config:2026.1.1-4444155|
| | | | | | | pf9-cindervolume-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-glance-role:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| [HOSTNAME2] | 172.29.32.110, 192.168.122.1, 10.0.11.102 | [HOST2_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SUCCESS getting host statuses...
```
{% endtab %}
{% endtabs %}
#### Step 2: Upgrade Hosts
To upgrade the hosts in a specific region, execute the following commands on the management cluster node:
{% hint style="info" %}
**NOTE**
The `upgrade-hosts` command times out after **1800 seconds (30 minutes)** by default. You can override this default by adding the `hostupgradetimeout` parameter in the `airctl-config.yaml` file. To prevent timeouts when upgrading a large number of hosts, increase this value proportionally — for every **50 hosts**, add **30 minutes (1800 seconds)** to the `hostupgradetimeout` value.
{% endhint %}
```bash
airctl upgrade-hosts --region
```
The command triggers the creation of a `host-upgrade-xxxx` pod in the corresponding region namespace. You can monitor the upgrade progress or verify its success by checking the pod status and logs:
```bash
# List the host upgrade pods
kubectl get pods -n | grep host-upgrade
# View logs of the host upgrade pod
kubectl logs -n
```
Here is a sample output:
{% tabs %}
{% tab title="Example" %}
```bash
host-upgrade-1747919688-g5rk5 5/5 Completed 0 83m
kubectl logs -n example-onprem-region1 host-upgrade-1747919688-g5rk5
```
{% endtab %}
{% endtabs %}
If a host fails to upgrade, run the following command to rerun the upgrade that the user executes using **IP.**
```bash
airctl upgrade-hosts --region --host-ips ""
```
#### Step 3: Verify Upgrade Status of the Hosts
After the upgrade, confirm that the host packages have been successfully updated by running the following command.
{% tabs %}
{% tab title="Example" %}
```bash
$ airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml
Getting host statuses...
Getting host statuses for region: [REGION_NAME]
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| Host Name | IP Addresses | Host ID | Host Agent Version | Status | Agent Status | Apps |
| [HOSTNAME1] | 172.29.32.34, 192.168.122.1, 10.0.11.10 | [HOST1_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-cindervolume-config:2026.1.1-4444155 |
| | | | | | | pf9-cindervolume-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-glance-role:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| [HOSTNAME2] | 172.29.32.110, 192.168.122.1, 10.0.11.102 | [HOST2_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SUCCESS getting host statuses...
```
{% endtab %}
{% endtabs %}
Compare the output with the previous recording to ensure the packages were updated as expected.
## Recovery: Upgrade Failure
#### Step 1. Stop the Failed Management Cluster
```bash
airctl stop --verbose
```
This command shuts down all components of the existing management cluster. `--verbose` helps verify the detailed output of all stopped services.
#### Step 2. Delete the Management Cluster Configuration
```bash
airctl unconfigure-du --force --verbose
```
This command removes all existing configuration files and metadata associated with the management cluster. Using `--force` enables overriding any locked or incomplete states.
#### Step 3: Delete the Existing Management Cluster
```bash
airctl delete-cluster --verbose
```
This command permanently deletes the management cluster resources. Ensure that you have backed up all critical data before running this command
#### Step 4. Create a New Management Cluster
```bash
airctl --config /opt/pf9/airctl/conf/airctl-config.yaml create-cluster --verbose
```
The command creates a new management cluster with the same configuration as before
#### Step 5. Restore from the Backup
```bash
airctl restore --backupdir --verbose
```
Replace ``with the actual path of your stored backup. The command restores the environment from the backup.
{% hint style="warning" %}
**Warning**
After a management plane upgrade, a rollback to a pre-upgrade version cannot be performed if any hosts have been upgraded as well.
{% endhint %}