# Upgrade This document provides a high-level overview of processes that are involved during the Platform9 Edge Cloud (PEC) on-premises upgrade process. This upgrade process is very similar to the PMK Installation process and follows similar steps except for a few other airgap specific processes. ## Upgrade Process The upgrade process is broken down into four main steps. 1. **Download & Install** — The PEC stack upgrade entails running the *install.sh* script, which upgrades the binaries and the associated components. 2. **DU upgrade** — The DU upgrade is accomplished via the newly upgraded ***airctl*** using the `upgrade-du` command.. 3. **HostAgent upgrade** — This is actually a two-part process 1. The first part is to clean up the (docker and rpm) caches of all existing hosts via the `configure-hosts` command. 2. The second part upgrades the host agent software on all hosts listed in the config file. (see details below) 4. **Cluster upgrade** — This update is executed via an API upgrade on each cluster. ### Download and Install This is similar to the Installation, you can download a newer version and run install exactly the way you would do that during install. {% tabs %} {% tab title="Bash" %} ```bash wget https://raw.githubusercontent.com/platform9/support-locker/master/edge-stack/download.sh chmod +x ./download.sh #./download.sh # in the following example super-secret-word is the secret provided by Platform9 team # v-5.6.0-xxxxx is the version to download and install ./download.sh super-secret-word v-5.6.0-xxxxx chmod +x ./install.sh ./install.sh v-5.6.0-xxxxx ``` {% endtab %} {% endtabs %} ## Upgrade Management Cluster This does not typically happen, but if there is an upgrade to the underlying management cluster, you may upgrade your existing cluster with the following steps. In your cluster spec file, update the nodelet package path to point to the latest version. {% tabs %} {% tab title="Bash" %} ```bash nodeletPkg: /opt/pf9/airctl/nodelet/nodelet.tar.gz ``` {% endtab %} {% endtabs %} Once set, run the following command to upgrade the cluster: {% tabs %} {% tab title="Bash" %} ```bash airctl --config /opt/pf9/airctl/conf/airctl-config.yaml advanced-ddu upgrade-mgmt ``` {% endtab %} {% endtabs %} ## Preload Images If the previous step has **NOT** been run, you need to ensure that containerd has access to the latest nodelet and kubedu image bundles. Copy the latest nodelet and kubedu images from /opt/pf9/airctl/imgs to /var/opt/pf9/images on all nodes and then restart the pf9-nodeletd service on each node. If there are multiple master nodes, this needs to be done in a staggered manner so as to not disrupt the API server availability. {% hint style="info" %} **Info** **Make sure to copy the nodelet and kubedu bundles that have the same version tag as airctl version.** {% endhint %} Once the images are copied over, restart the pf9-nodeletd service and wait until all the phases report running. This needs to be done on all nodes of the management cluster. {% tabs %} {% tab title="Bash" %} ```bash systemctl restart pf9-nodeletd ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Bash" %} ```bash INDEX NUMBER FILE NAME PHASE STATUS 1 Generate certs / Send signing request to CA running 2 Prepare configuration running 3 Configure Container Runtime running 4 Start Container Runtime running 5 Load user images to container runtime running 6 Configure etcd running 7 Start etcd running 8 Network configuration running 9 Configure CNI plugin running 10 Configure and start pf9-bouncer running 11 Miscellaneous scripts and checks running 12 Configure and start kubelet running 13 Configure and start kube-proxy running 14 Wait for k8s services and network to be up running 15 Apply and validate node taints running 16 Apply dynamic kubelet configuration running 17 Uncordon node running 18 Deploy app catalog running 19 Configure and start Keepalived running 20 Configure and start coredns running 21 Drain all pods (stop only operation) running ``` {% endtab %} {% endtabs %} It takes a while for these images to be loaded into containerd. Please run the following command to check if the images with the new tags are present on the host. You can proceed to the next step once the new images show up. {% tabs %} {% tab title="Bash" %} ```bash /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io images ``` {% endtab %} {% endtabs %} ## Upgrade Management Plane Run the *configure-localhost* command first to make sure that IPv4\*\*/\*\*IPv6 settings are configured, and the HAProxy configuration is initialized. {% tabs %} {% tab title="Bash" %} ```bash airctl upgrade --config ``` {% endtab %} {% endtabs %} The next task after the installation is to upgrade the DU. ### Configure Host and HostAgent Upgrade Platform9 uses a two-step upgrade process. The first step is to run a *configure-hosts* command to update any yum-repos or add docker images (this is being deprecated and moving to a centralized registry for images and a centralized yum-repo). This step is also needed if you want to change the DU Name to IP mapping. {% hint style="info" %} **Info** Before running host upgrade, ensure the *airct-config.yaml* file has the correct file names set in the *hostAgentRepo* and *dockerRepo* parameters. {% endhint %} To upgrade the nodes, run the *configure-hosts* command followed by the *upgrade-hosts* command. This will only upgrade nodes listed in the *airctl-config.yaml.* {% hint style="info" %} **Info** This will only upgrade nodes listed in the *airctl-config.yaml.* This allows you to upgrade only a subset of the nodes at a time. {% endhint %} {% tabs %} {% tab title="Bash" %} ```bash /opt/pf9/airctl/airctl configure-hosts [airctl-1-1672253-128.pf9.localnet] | HOST | SUCCESS | MESSAGE | [airctl-1-1672253-128.pf9.localnet] |-----------------------------------------------|---------|---------| [airctl-1-1672253-128.pf9.localnet] | 10.128.147.243 | true | | [airctl-1-1672253-128.pf9.localnet] | 10.128.146.10 | true | | [airctl-1-1672253-128.pf9.localnet] | test-pf9-rbk-c7-pmk-a-1672253-400-1.novalocal | true | | ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Bash" %} ```bash /opt/pf9/airctl/airctl upgrade-hosts success ``` {% endtab %} {% endtabs %} ### Host Status You can obtain the status of each host's kubernetes version and hostagent version before you proceed to the next step using the following command. {% tabs %} {% tab title="Bash" %} ```bash /opt/pf9/airctl/airctl host-status DU hostagent version: 5.3.0.3772.a458a6d Host IP UUID Hostagent Kube Version Status ---- ---- ---- --------- ------------ ------ t-3 10.128.147.73 46eb0159-58d7-46fb-bbc0 5.3.0-3772 1.20.5-pmk.1917 ok t-2 10.128.147.185 5a2c84eb-6ca4-41b3-90a9 5.3.0-3772 1.20.5-pmk.1917 ok t-1 10.128.146.212 dfa46c4f-0e23-41a8-a74e 5.3.0-3772 1.20.5-pmk.1917 ok ``` {% endtab %} {% endtabs %} Now you are ready to upgrade your cluster whenever you would like. ### Upgrade Cluster You can also upgrade the cluster upgrade via the [Qbert API.](https://platform9.com/docs/qbert/ref#postupgrade-a-cluster-identified-by-the-uuid) There may be other specific considerations for upgrading the cluster, which can be found on the [upgrade your cluster](https://platform9.com/docs/kubernetes/upgrade-cluster#upgrade-your-cluster) page. ### API For Host Data In the DU shell, users can obtain relevant data by querying the hostagent on the node by running *REST API GET at* [***http://localhost:8082/v1/hosts/\*** and noting the `host_agent` section in the output. The version field represents the hostagent version that is currently running, while the status field has one of two values, either *running*`or` *updating*. The default status is `_running_,`while the *updating* state indicates that it has received the command and is updating and processing it. If the host reports offline, the hostagent is down or connectivity to the host is broken. {% tabs %} {% tab title="Bash" %} ```bash curl -s -H "X-Auth-Token: $TOK" http://localhost:8082/v1/hosts/f72dd9ca-06ca-4846-be21-833b7edd8336 | jq .host_agent { "status": "running", "version": "5.1.0-3534.991bd2f" } ``` {% endtab %} {% endtabs %} Additionally, users can upgrade the hostagent on the host using the REST API to invoke a PUT via [***http://localhost:8082/v1/hosts/\/hostagent*** using a request body containing the name, version, and URL of the node for the hostagent package. {% tabs %} {% tab title="Bash" %} ```bash curl -s -H "X-Auth-Token: $TOK" -X PUT http://localhost:8082/v1/hosts/f72dd9ca-06ca-4846-be21-833b7edd8336/hostagent --data '{"name":"hostagent", "version":"5.2.0-3788-289b32e", "url": "http://localhost:9080:private/pf9-hostagent-5.2.0-3788-289b32e.rpm"}' ``` {% endtab %} {% endtabs %} ## Rollback ### Considerations * Any cluster created on the DU after the DU upgrade will cease to communicate with the DU after the DU is rolled back. Such clusters will have to be destroyed. Nodes will have to be cleaned up and then connected to the rolled back DU before bootstrapping a fresh cluster. * Rollback was designed to be used shortly after a failed upgrade and is not intended as a general purpose backup/restore feature. ### How to * Use these steps to stop and unconfigure the KDU. **Info** These commands may or may not fail, depending on the actual state of the failed upgraded KDU. {% tabs %} {% tab title="Bash" %} ```bash $ airctl stop ``` {% endtab %} {% endtabs %} * To roll back to the previous KDU, use this command. {% tabs %} {% tab title="Bash" %} ```bash airctl rollback --dir /tmp/airctl_backup_156579090 ``` {% endtab %} {% endtabs %} * To start the KDU, use this command. {% tabs %} {% tab title="Bash" %} ```bash $ airctl --config /opt/pf9/airctl/conf/airctl-config.yaml advanced-ddu scale-du up ``` {% endtab %} {% endtabs %} * Run validations * The Kubernetes nodes and clusters should show as connected. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.platform9.com/self-managed-cloud-platform/5.7/upgrade.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.