# Upgrade Guide: PEC 5.6 to SMCP 5.9.1

## Prerequisites

Before beginning the upgrade process, ensure the following requirements are met:

1. Ensure that the [Minimum Requirements](https://platform9.com/docs/v5.9/PEC/requirements) are met.
2. Minimum Kubernetes version for Workload Cluster should be `v1.22`.

## Upgrade Procedure

### 1. Backup Current System

Follow the [Backup Guide](https://platform9.com/docs/v5.9/PEC/backup-guide) to back up the `5.6` management plane. Before proceeding, verify that the backup was created successfully.

### 2. Take etcd Snapshot

Before making any changes to the system, it's crucial to take a snapshot of the `etcd` cluster. This provides an additional recovery point if needed.

**a**. Check the `etcd` cluster status and identify the leader node:

{% tabs %}
{% tab title="Bash" %}

```bash
sudo /opt/pf9/pf9-kube/bin/etcdctl endpoint status --cluster -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"
```

{% endtab %}
{% endtabs %}

Make a note of which node is the leader.

**b**. Take the `etcd` cluster backup:

{% tabs %}
{% tab title="None" %}

```none
sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot save <path/backupfile.db> --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --endpoints=https://<master_IP>:4001
```

{% endtab %}
{% endtabs %}

Replace `<path/backupfile.db>` with the desired path and filename for your backup and `<master_IP>` with the IP address of the master node.

**c**. Validate the `etcd` backup:

{% tabs %}
{% tab title="None" %}

```none
sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot status --write-out=table <path/backupfile.db>
```

{% endtab %}
{% endtabs %}

Ensure that the snapshot status shows as valid before proceeding.

### 3. Destroy Old Cluster

Remove the existing `1.22` based nodelet cluster:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl stop
# Stops all running services managed by airctl

airctl unconfigure-du --force 
# Removes DU configuration and forcefully cleans up resources

airctl advanced-ddu delete-mgmt
# Deletes the management plane components
```

{% endtab %}
{% endtabs %}

### 4. Clean Up Configuration

On all master nodes, clean up the certificate directory:

{% tabs %}
{% tab title="Bash" %}

```bash
rm -rf /etc/nodelet/airctl-mgmt/certs/*
# This command removes all files in the specified directory
```

{% endtab %}
{% endtabs %}

Verify the directory is empty before proceeding.

### 5. Download and Install New Artifacts

To obtain and install the SMCP 5.9.1 artifacts:

**a.** Download the download script:

{% tabs %}
{% tab title="None" %}

```none
wget https://raw.githubusercontent.com/platform9/support-locker/master/edge-stack/download.sh
```

{% endtab %}
{% endtabs %}

**b.** Make the script executable:

{% tabs %}
{% tab title="None" %}

```none
chmod u+x download.sh
```

{% endtab %}
{% endtabs %}

**c.** Run the download script:

{% tabs %}
{% tab title="None" %}

```none
./download.sh <PF9-SHARED-SECRET> v-5.9.1-<airctl_version>
```

{% endtab %}
{% endtabs %}

Replace `<PF9-SHARED-SECRET>` with your Platform9 shared secret.

**d.** After downloading the artifacts, verify their integrity using the following command:

{% tabs %}
{% tab title="Bash" %}

```bash
curl --user-agent <secret> https://pf9-airctl.s3-accelerate.amazonaws.com/<version>/md5sum.txt | md5sum -c -
```

{% endtab %}
{% endtabs %}

Replace `<secret>` with the same one used to download the `airctl` build from S3 and `<version>` with the SMCP version you're installing (e.g., `v5.9.1`).

**e.** Run the `install.sh` script to extract and place the artifacts under `/opt/pf9/airctl` directory.

{% tabs %}
{% tab title="Bash" %}

```bash
bash install.sh v-5.9.1-<airctl_version>
```

{% endtab %}
{% endtabs %}

Refer to the [SMCP 5.9 release notes](https://github.com/platform9/pcd-docs-gitbook/blob/main/docs/v5.9/PEC/lts3-release-notes/README.md) for the exact `airctl` version used by the `install.sh` script.

For example, when upgrading to SMCP `v5.9.1` :

{% tabs %}
{% tab title="example" %}

```bash
bash install.sh v-5.9.1-3097398
```

{% endtab %}
{% endtabs %}

### 6. Configure New Management Cluster

Update configuration files before deployment:

1. Edit `airctl-config.yaml`:
   1. Update `hostAgentRepo` to use SMCP 5.9.1 images.
   2. Update `dockerRepo` to use SMCP 5.9.1 images.
2. Edit `nodelet-bootstrap.yaml`:
   1. Update `systemImages` to use SMCP 5.9.1 image files
   2. Set `userImages` to point to PEC 5.6.7 image files
   3. Add `isAirgapped: true` for offline installations

Example `airctl-config.yaml` and `nodelet-bootstrap.yaml` snippet:

{% tabs %}
{% tab title="airctl-config.yaml" %}

```none
hostAgentRepo: /opt/pf9/airctl/hostagent-8-v-5.9.1-<build_number>.tar.gz
dockerRepo: /opt/pf9/airctl/docker-8-v-5.9.1-<build_number>.tar.gz
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="nodelet-bootstrap.yaml" %}

```yaml
isAirgapped: true
systemImages:
- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.9.1-<build_number>.tar.gz
- /opt/pf9/airctl/imgs/nodelet-imgs-v-5.9.1-<build_number>.tar.gz
userImages:
- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.6.7-<build_number>.tar.gz
```

{% endtab %}
{% endtabs %}

### 7. Deploy New Management Cluster

Deploy the new `1.25` based nodelet management cluster:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl advanced-ddu create-mgmt --verbose
# This command creates and deploys the new management plane components
```

{% endtab %}
{% endtabs %}

Initialize the management plane:

{% tabs %}
{% tab title="None" %}

```none
sudo airctl init
```

{% endtab %}
{% endtabs %}

Monitor the deployment progress and check for any errors.

### 8. Restore from Backup

Restore the PEC `5.6` backup by following the [restore guide](https://github.com/platform9/pcd-docs-gitbook/blob/main/docs/v5.9/PEC/restore-guide/README.md) for the `5.9.1` management plane on top of the `1.25` nodelet cluster.

{% hint style="warning" %}
**Consul Issue:**

After restoring, you might see errors in the Consul pod saying `ACL not found`. These errors are harmless and can be ignored.

Therefore, SMCP currently skips the Consul upgrade in the upgrade step and installs the upgraded Consul during the restore process itself.
{% endhint %}

### 9. Verify DU Status

Check that the DU is in a ready state:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl status
# This command displays the current status of the DU
```

{% endtab %}
{% endtabs %}

Wait until all components show as ready before proceeding.

### 10. Upgrade DU Components

To upgrade the DU components to`5.9.1`, follow the [upgrade guide](https://github.com/platform9/pcd-docs-gitbook/blob/main/docs/v5.9/PEC/Upgrade-guide/README.md).

{% hint style="info" %}
**NOTE**

To prevent potential ImagePullBackOff errors, refer to [additional-information](https://platform9.com/kb/PEC/while-upgrade-from-lts2-patch2-to-scmp--management-plane-cluster#additional-information)
{% endhint %}

After upgrade, verify DU status again:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl status
```

{% endtab %}
{% endtabs %}

### 11. Verify Upgrade in GUI

Log in to the management UI and confirm the version is displayed as `5.9.1`.

### 12. Push New Kubernetes Images

Push images for Kubernetes **1.24**, **1.25**, and **1.26** to your private registry (if applicable):

{% tabs %}
{% tab title="Bash" %}

```bash
sudo airctl advanced-du push-images --img-tar [tar file name]
```

{% endtab %}
{% endtabs %}

To push all new images at once:

{% tabs %}
{% tab title="Bash" %}

```bash
ls -1 /opt/pf9/airctl/imgs/k8s* | xargs -I{} sh -c "sudo /opt/pf9/airctl/airctl --config /opt/pf9/airctl/conf/airctl-config.yaml advanced-du push-images --img-tar {}"
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**NOTE**

If the **privateRegistryType** value is **custom,** the images must be manually pushed to the custom registry.
{% endhint %}

### 13. Upgrade Host Components

Upgrade the host-side components:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl configure-hosts --reset-pf9-managed-docker-conf --skip-docker-install --skip-docker-img-import --verbose
# This command configures hosts with new settings

airctl hosts-status
# Check the status of hosts before proceeding

airctl upgrade-hosts --verbose
# This command upgrades the host components
```

{% endtab %}
{% endtabs %}

### 14. Verify Status and Version

After upgrading, verify management plane services are ready. Run the following command to verify.

{% tabs %}
{% tab title="Bash" %}

```bash
airctl status
# Check the status of management plane

airctl version
# The output should be: v-5.9.1-3097398

kubectl get nodes -o wide
# Ensure all the nodes are in Ready state
```

{% endtab %}
{% endtabs %}

At this point all the components have been upgraded to SMCP `5.9.1` (LTS3).

Now, proceed with the actual workload cluster upgrades either using the [GUI](https://github.com/platform9/pcd-docs-gitbook/blob/main/kubernetes/upgrade-cluster/README.md) or [API](https://platform9.com/docs/qbert#postupgrade-a-cluster-identified-by-the-uuid).