Upgrade Guide: PEC 5.6 to SMCP 5.9.1

Prerequisites

Before beginning the upgrade process, ensure the following requirements are met:

1. Ensure that the Minimum Requirements are met.

2. Minimum Kubernetes version for Workload Cluster should be v1.22.

Upgrade Procedure

1. Backup Current System

Follow the Backup Guide to back up the 5.6 management plane. Before proceeding, verify that the backup was created successfully.

2. Take etcd Snapshot

Before making any changes to the system, it's crucial to take a snapshot of the etcd cluster. This provides an additional recovery point if needed.

a. Check the etcd cluster status and identify the leader node:

Bash

sudo /opt/pf9/pf9-kube/bin/etcdctl endpoint status --cluster -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

Make a note of which node is the leader.

b. Take the etcd cluster backup:

None

sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot save <path/backupfile.db> --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --endpoints=https://<master_IP>:4001

Replace <path/backupfile.db> with the desired path and filename for your backup and <master_IP> with the IP address of the master node.

c. Validate the etcd backup:

None

sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot status --write-out=table <path/backupfile.db>

Ensure that the snapshot status shows as valid before proceeding.

3. Destroy Old Cluster

Remove the existing 1.22 based nodelet cluster:

Bash

airctl stop
# Stops all running services managed by airctl

airctl unconfigure-du --force 
# Removes DU configuration and forcefully cleans up resources

airctl advanced-ddu delete-mgmt
# Deletes the management plane components

4. Clean Up Configuration

On all master nodes, clean up the certificate directory:

Bash

rm -rf /etc/nodelet/airctl-mgmt/certs/*
# This command removes all files in the specified directory

Verify the directory is empty before proceeding.

5. Download and Install New Artifacts

To obtain and install the SMCP 5.9.1 artifacts:

a. Download the download script:

None

wget https://raw.githubusercontent.com/platform9/support-locker/master/edge-stack/download.sh

b. Make the script executable:

None

chmod u+x download.sh

c. Run the download script:

None

./download.sh <PF9-SHARED-SECRET> v-5.9.1-<airctl_version>

Replace <PF9-SHARED-SECRET> with your Platform9 shared secret.

d. After downloading the artifacts, verify their integrity using the following command:

Bash

curl --user-agent <secret> https://pf9-airctl.s3-accelerate.amazonaws.com/<version>/md5sum.txt | md5sum -c -

Replace <secret> with the same one used to download the airctl build from S3 and <version> with the SMCP version you're installing (e.g., v5.9.1).

e. Run the install.sh script to extract and place the artifacts under /opt/pf9/airctl directory.

Bash

bash install.sh v-5.9.1-<airctl_version>

Refer to the SMCP 5.9 release notes for the exact airctl version used by the install.sh script.

For example, when upgrading to SMCP v5.9.1 :

example

bash install.sh v-5.9.1-3097398

6. Configure New Management Cluster

Update configuration files before deployment:

1. Edit airctl-config.yaml:

   1. Update hostAgentRepo to use SMCP 5.9.1 images.

   2. Update dockerRepo to use SMCP 5.9.1 images.

2. Edit nodelet-bootstrap.yaml:

   1. Update systemImages to use SMCP 5.9.1 image files

   2. Set userImages to point to PEC 5.6.7 image files

   3. Add isAirgapped: true for offline installations

Example airctl-config.yaml and nodelet-bootstrap.yaml snippet:

airctl-config.yaml

hostAgentRepo: /opt/pf9/airctl/hostagent-8-v-5.9.1-<build_number>.tar.gz
dockerRepo: /opt/pf9/airctl/docker-8-v-5.9.1-<build_number>.tar.gz

nodelet-bootstrap.yaml

isAirgapped: true
systemImages:
- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.9.1-<build_number>.tar.gz
- /opt/pf9/airctl/imgs/nodelet-imgs-v-5.9.1-<build_number>.tar.gz
userImages:
- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.6.7-<build_number>.tar.gz

7. Deploy New Management Cluster

Deploy the new 1.25 based nodelet management cluster:

Bash

airctl advanced-ddu create-mgmt --verbose
# This command creates and deploys the new management plane components

Initialize the management plane:

None

sudo airctl init

Monitor the deployment progress and check for any errors.

8. Restore from Backup

Restore the PEC 5.6 backup by following the restore guide for the 5.9.1 management plane on top of the 1.25 nodelet cluster.

Consul Issue:

After restoring, you might see errors in the Consul pod saying ACL not found. These errors are harmless and can be ignored.

Therefore, SMCP currently skips the Consul upgrade in the upgrade step and installs the upgraded Consul during the restore process itself.

9. Verify DU Status

Check that the DU is in a ready state:

Bash

airctl status
# This command displays the current status of the DU

Wait until all components show as ready before proceeding.

10. Upgrade DU Components

To upgrade the DU components to5.9.1, follow the upgrade guide.

NOTE

To prevent potential ImagePullBackOff errors, refer to additional-information

After upgrade, verify DU status again:

Bash

airctl status

11. Verify Upgrade in GUI

Log in to the management UI and confirm the version is displayed as 5.9.1.

12. Push New Kubernetes Images

Push images for Kubernetes 1.24, 1.25, and 1.26 to your private registry (if applicable):

Bash

sudo airctl advanced-du push-images --img-tar [tar file name]

To push all new images at once:

Bash

ls -1 /opt/pf9/airctl/imgs/k8s* | xargs -I{} sh -c "sudo /opt/pf9/airctl/airctl --config /opt/pf9/airctl/conf/airctl-config.yaml advanced-du push-images --img-tar {}"

NOTE

If the privateRegistryType value is custom, the images must be manually pushed to the custom registry.

13. Upgrade Host Components

Upgrade the host-side components:

Bash

airctl configure-hosts --reset-pf9-managed-docker-conf --skip-docker-install --skip-docker-img-import --verbose
# This command configures hosts with new settings

airctl hosts-status
# Check the status of hosts before proceeding

airctl upgrade-hosts --verbose
# This command upgrades the host components

14. Verify Status and Version

After upgrading, verify management plane services are ready. Run the following command to verify.

Bash

airctl status
# Check the status of management plane

airctl version
# The output should be: v-5.9.1-3097398

kubectl get nodes -o wide
# Ensure all the nodes are in Ready state

At this point all the components have been upgraded to SMCP 5.9.1 (LTS3).

Now, proceed with the actual workload cluster upgrades either using the GUI or API.