# Upgrade Management Plane and Hosts ### Upgrade Management Plane and Hosts The upgrade of the self-hosted space.vars.product\_acronym is a two-phase, sequential process: * **Management Plane:** Upgrading the Management Plane updates core services, management APIs, region-specific configurations, and orchestration components. * **Hosts**: Once the management plane is upgraded, hosts in each region must be upgraded to ensure compatibility, seamless communication, and completion of the overall upgrade process. Before you begin the upgrade, ensure you meet all prerequisites. ### Prerequisites for Upgrade {% hint style="info" %} NOTE Review the Upgrade Notes for the [Broken mention](broken://pages/OrCDIsPYuwbp1AS9XXWn). {% endhint %} #### Verify Management Plane Health Before starting the upgrade, make sure the management plane health is in a good state. Run `airctl status` and check that the *Region Health* shows as **Ready**. If it's ready, you can proceed with the upgrade. If not, look into the affected regions, fix any issues, and resolve problems such as pods stuck in a `CrashLoopBackOff` state. #### Manual Backup You must perform a [Backup and Restore Management Plane](/private-cloud-director/2025.7/getting-started/self-hosted/backup-and-restore.md#important-considerations) of your management plane infrastructure before starting the upgrade. This backup is critical for recovery. ### Upgrade Management Plane and Cluster Upgrade your space.vars.product\_acronym environment to access the latest features, security patches, and performance improvements through this two-step process. #### Upgrade Management Plane Updating the Management Plane running on the Management Cluster ensures multiple components are automatically upgraded as a part of an `airctl upgrade` process. Here are the components that will be affected. * Core services and applications. * Management APIs and user interfaces. * Region-specific configurations and services. * Service coordination and orchestration components. #### **The Impact of the Management Plane Upgrade** {% hint style="info" %} **NOTE** Temporary service interruptions may occur during the upgrade process. {% endhint %} * The user interface will be unavailable during the upgrade. * All API services will be temporarily unavailable. * Create, Read, Update, and Delete operations will not be possible during this time. * **Running workloads on VMs will not be impacted.** #### **Step 1: Download and Run Installer Artifacts** `airctl` is the command-line installer for space.vars.product\_name. Run the following commands **only on a verified management cluster node** to download `airctl` along with the required installation artifacts. 1. Download the Installer Script and Artifacts using the following command. ```bash curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/latest/index.txt | awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/latest/" $NF "\" -o ${HOME}/" $NF}' | bash ``` You can choose to download a specific version of the installer script and artifact. For example, you can replace `latest` with a specific version, such as `v-2025.7.0-3990314`. Here is a sample modified command: ```bash curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/v-2025.7.0-3990314/index.txt | \ awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/v-2025.7.0-3990314/" $NF "\" -o ${HOME}/" $NF}' | bash ``` {% hint style="info" %} **NOTE** Replace `` with your user key. {% endhint %} 2. Make the Installer Executable using the following command. ```bash chmod +x ./install-pcd.sh ``` 3. Run the Installation Script using the following command. ```bash ./install-pcd.sh `cat version.txt` ``` This installs the specific version of `airctl` based on the `version.txt` file. {% hint style="info" %} **NOTE** It is recommended to run a `airctl status` to ensure all the `desired services` are equal to `ready services` before you begin the upgrade. {% endhint %} 4. Add symbolic link for `airctl` to the system path so that it can be invoked without specifying full path. Also, create a symbolic link to the `airctl-config.yaml` file in the users home directory so that the configuration file need not be specified for every `airctl` invocation. #### **Proxy Configuration (Optional)** If your environment uses a network proxy, update the values `/opt/pf9/airctl/conf/helm_values/kplane.template.yml` again, as the changes would be lost after a new build is installed. Update it again as shown below: ```bash cat /opt/pf9/airctl/conf/helm_values/kplane.template.yml | grep proxy # Sample output: https_proxy: "http://squid.platform9.horse:3128" http_proxy: "http://squid.platform9.horse:3128" no_proxy: "10.149.106.11,10.149.106.12,10.149.106.13,10.149.106.14,10.149.106.15,10.149.106.16,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.platform9.localnet,.default.svc" ``` The list of IP addresses in the `no_proxy` list should include the master-ips, worker-ips, external-ip4, master-vip4, and any other addresses for which traffic should not be routed via a proxy server. {% hint style="info" %} NOTE The `upgrade-cluster` step is skipped in this release, as the nodelet Kubernetes version remains unchanged at **v1.30**. {% endhint %} #### **Step 2: Upgrade all regions** To upgrade **all** regions set up on your Private Cloud Director, execute the following command. ```bash airctl upgrade ``` **Optionally**, you can upgrade a specific region by replacing `` with your target region. ```bash airctl upgrade --region ``` Here is the modified sample command. {% tabs %} {% tab title="Example" %} ```bash # airctl upgrade --region Region1 INFO rollback state directory /tmp/airctl_ddu_backup_Region1_240020739 INFO Saving the helm revisions to the state file INFO --- backing up region--- Region1 INFO Archive created successfully for Region1. Backup backup.tar.gz saved to /tmp/airctl_ddu_backup_Region1_240020739 INFO --- moving old state file to /tmp/airctl_ddu_backup_Region1_240020739/state.yaml --- INFO --- moving old kplane_values.yaml file to /tmp/airctl_ddu_backup_Region1_240020739/kplane_values.yaml --- SUCCESS Upgrading region Region1 upgrade done ``` {% endtab %} {% endtabs %} To monitor and diagnose the upgrade logs, add `--verbose`. ```bash airctl upgrade --verbose ``` ### **Verify: Upgrade Success on Management Plane** After completing the upgrade of the management plane, verify that you can continue to access the user interface, then verify the upgrade's success by checking the deployment status across all regions. Run the following command: ```bash airctl status ``` Here is the sample output. {% tabs %} {% tab title="Example" %} ```bash ------------- deployment details --------------- fqdn: airctl-1-4053079-898.platform9.localnet region: airctl-1-4053079-898 deployment status: ready region health: ✅ Ready version: PCD 2025.7-47 -------- region service status ---------- desired services: 30 ready services: 30 ------------- deployment details --------------- fqdn: airctl-1-4053079-898-blr.platform9.localnet region: airctl-1-4053079-898-blr deployment status: ready region health: ✅ Ready version: PCD 2025.7-47 -------- region service status ---------- desired services: 81 ready services: 81 ------------- deployment details --------------- fqdn: airctl-1-4053079-898-hkg.platform9.localnet region: airctl-1-4053079-898-hkg deployment status: ready region health: ✅ Ready version: PCD 2025.7-47 -------- region service status ---------- desired services: 81 ready services: 81 ``` {% endtab %} {% endtabs %} After a successful upgrade and verification, space.vars.product\_acronym does not support rollback to a previous version. ### Upgrade the Hosts After the management plane is successfully upgraded, the hosts running on each of your regions must be updated. The agents running on the host manage communication between hosts and the management plane. #### The Impact from Hosts Upgrade * No service unavailability is expected during the host upgrade. * Workloads running on VMs will continue to operate without disruption. #### **Step 1: Record the Current Version of Packages on Host** Before you begin the upgrade, execute the following command on each onboarded host to record the current package versions: ```bash dpkg -l | grep pf9 ``` #### **Step 2: Upgrade Hosts** To upgrade the hosts in a specific region, execute the following commands on the management cluster node: ```bash export CONSUL_HTTP_ADDR=http://decco-consul-consul-ui.default.svc.cluster.local export CONSUL_HTTP_TOKEN=$(grep consulToken ${HOME}/.airctl/state.yaml | cut -d' ' -f2) /opt/pf9/airctl/kpctl -cmd upgrade-hosts -fqdn ``` This triggers the creation of a `host-upgrade-xxxx` pod in the corresponding region namespace. You can monitor the upgrade progress or verify its success by checking the pod status and logs: ```bash # List the host upgrade pods kubectl get pods -n | grep host-upgrade # View logs of the host upgrade pod kubectl logs -n ``` Here is a sample output: {% tabs %} {% tab title="Example" %} ```bash host-upgrade-1747919688-g5rk5 5/5 Completed 0 83m kubectl logs -n example-onprem-region1 host-upgrade-1747919688-g5rk5 ``` {% endtab %} {% endtabs %} If a host fails to upgrade, execute the following command to re-run the upgrade using its **host ID** (found in the pod logs): ```bash /opt/pf9/airctl/kpctl -cmd upgrade-hosts -fqdn -host-ids ``` #### **Step 3: Verify Upgrade Status of the Hosts** After the upgrade, confirm that the host packages have been successfully updated by running the following command again on each host: ```bash dpkg -l | grep pf9 ``` Compare the output with the previous recording to ensure the packages were updated as expected. ### Recovery: Upgrade Failure #### **Step 1. Stop the Failed Management Cluster** ```bash airctl stop --verbose ``` This command shuts down all components of the existing management cluster. `--verbose` helps verify the detailed output of all stopped services. #### **Step 2. Delete the Management Cluster Configuration** ```bash airctl unconfigure-du --force --verbose ``` This command removes all existing configuration files and metadata associated with the management cluster. Using `--force` enables overriding any locked or incomplete states. #### **Step 3: Delete the Existing Management Cluster** ```bash airctl delete-cluster --verbose ``` This command permanently deletes the management cluster resources. Ensure that you have backed up all critical data before running this command. #### **Step 4. Create a New Management Cluster** ```bash airctl --config /opt/pf9/airctl/conf/airctl-config.yaml create-cluster --verbose ``` This command creates a new management cluster with the same configuration as before. #### **Step 5. Restore from the Backup** ```bash airctl restore --backupdir --verbose ``` Replace `` with the actual path of your stored backup. This command restores the environment from the backup. {% hint style="warning" %} **Warning** After a management plane upgrade, a rollback to a pre-upgrade version cannot be performed if any hosts have been upgraded as well. {% endhint %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.platform9.com/private-cloud-director/2025.7/getting-started/self-hosted/upgrade.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.