Cluster Scaling & Other Operations

This document describes steps to scale your Kubernetes management cluster that is part of your self-hosted Private Cloud Director deployment.

Scale Up Management Cluster

Let's consider that we have 3 master nodes as part of your management cluster, with IP addresses 1.1.1.1, 2.2.2.2 `and` 3.3.3.3.

$ cat /opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml
...
masterNodes:
- nodeName: 1.1.1.1
- nodeName: 2.2.2.2
- nodeName: 3.3.3.3

Lets say that you want to scale up the management cluster to 5 nodes, and that 4.4.4.4 & 5.5.5.5 are the IP addresses of the 2 new nodes to be added.

To scale up the number of cluster nodes to 5:

  • Configure the pre-requisites on the two new nodes.

  • Edit the cluster bootstrap configuration file /opt/pf9/airctl/conf/nodelet-bootstrap-config.yamland add the two new IP addresses to the section called masterNodes in the file.

  • Finally, run the airctl command shown below to scale up the cluster.

Note

airctl expects the node count to be always odd in the cluster bootstrap configuration file /opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml

$ cat /opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml
...
masterNodes:
- nodeName: 1.1.1.1
- nodeName: 2.2.2.2
- nodeName: 3.3.3.3
- nodeName: 4.4.4.4
- nodeName: 5.5.5.5

Now verify that the management cluster has scaled up by querying the cluster nodes.

Management Cluster Status

To check the status of your management cluster, run the following command:

Sample output:

Scale Down Management Cluster

Now let's assume that we want to remove nodes 2.2.2.2 and 3.3.3.3 from the management cluster. To scale down the cluster:

  • Edit the cluster bootstrap configuration file and remove the IP addresses of the two nodes.

  • Then run the airctl command as shown below to scale down the cluster.

Warning

When we run above command to scale down the cluster, due to a known issue, it tries to remove the first node in this case 2.2.2.2 and fails due to a containerd mount cleanup issue. See the workaround below.

Workaround

We need to manually unmount the containerd partitions from the node to be removed.

Then, run command kubectl delete node <IPAddress> on the cluster after which all the terminating pods for that node move to other nodes part of the cluster.

It can take around 10-15 mins for all the pods from this node to move to the other nodes. Please wait and ensure this has happened before proceeding to next step.

  • Because the scale command errored out on the first node to be removed, run the scale command again to remove the second node. Also, ensure to perform the manual workaround steps again in this case for node 3.3.3.3

  • Cluster state post scale down operation completion:

Stop Management Plane/Regions

To stop the specific region of your self-hosted deployment, run the following command. If you want to stop all regions, just remove --region flag.

Sample output:

Start Management Plane/Regions

To start the specific region of your self-hosted deployment, run the following command. If you want to start all regions, just remove --region flag.

Sample output:

Uninstall Self-Hosted Deployment

To uninstall the specific region of your self-hosted deployment, run the following command. If you want to uninstall all regions, just remove --region flag.

This command will uninstall and remove configured regions along with all infrastructure software like consul, vault, percona, k8sniff, etc.

If you plan to reuse the same nodes to deploy a new self-hosed Private Cloud Director environment, make sure to also run the following command on all nodes first.

PreviousUpgrade Management Plane and HostsNextBackup and Restore Management Plane

Last updated

Was this helpful?