Dynamic Kubelet Configuration

If you wish to change the kubelet configuration on a live cluster, we can use the dynamic kubelet configuration feature provided by Platform9. The dynamic kubelet configuration saves us the effort of bringing down a live cluster to make changes to reconfigure kubelet. The standard Kubernetes feature for Dynamic Kubelet Configuration was removed from upstream Kubernetes starting version k8s 1.24.

Feature Support

This feature is supported for PMK 5.10.2 version and above.

On Platform9's Managed Kubernetes, separate default ConfigMaps are present for both master and worker nodes on the Management plane sunpike, named as worker-default-kubelet-config& master-default-kubelet-config

Users do not have a direct access to the management plane sunpike, and thus have to interact via Qbert API.

Using the dynamic kubelet configuration, you can incorporate changes to one or more parameters. All nodes running kubelet that use the configuration will detect the changes in the dynamic kubelet configurations and then integrate those changes into the ConfigMap settings on the kubernetes nodes and then restart the kubelet service. Once restarted, the kubelet service uses the new configuration in the ConfigMap.

You can find the actual kubelet configuration file as well on the nodes. The default kubelet file is located at /var/opt/pf9/kube/kubelet-config/bootstrap-config.yaml

Dynamic Kubelet Configuration using Qbert APIs

Viewing existing Kubelet Configurations

To view existing custom Kubelet configurations, use the Qbert Get Kubelet Config API. It will return configurations for all clusters under projectId. sampleKubeletConfig will have the reference to default configuration stored in sunpike.

Sample usage:

Bash

curl --request GET \
 --url 'https://platform9.com/qbert/v4/{projectId}/kubeletconfig' \
 --header 'X-Auth-Token: {X-Auth-Token}'

Sample Response:

JSON

{
  "sampleKubeletConfig": {
            "worker" : "base64-data",
            "master" : "base64-data"
  },
  "customKubeletConfig": 
  [
      {
              "clusterId": "f3f2ae5b-78b0-4c66-b5c1-75003eb5b6c3",
              "config": [
                      {
                              "configName" : "cm-1",
                              "configData" : "base64-config-data-string",
                              "role" : "worker",
                              "nodeIps" : ["10.10.10.10"]
                      },
                      {
                              "configName" : "cm-2",
                              "configData" : "base64-config-data-string",
                              "role" : "master",
                              "nodeIps" : ["10.10.10.12"]
                      }
              ]
      },
      {
              "clusterId": "abc2ae5b-78b0-4c66-b5c1-75003eb5bjqd",
              "config": [
                      {
                              "configName" : "cm-3",
                              "configData" : "base64-config-data-string",
                              "role" : "both",
                              "applyAll" : true,
                      }
              ]
      }
  ]
  
}

Creating new Kubelet Configurations

New Kubelet configurations can be created using the Qbert Post Kubelet Config API. The API lets you create new ConfigMaps in sunpike to store the new kubelet configurations. You can pass multiple configurations in one API call.

The configurations are stored in default namespace on the sunpike. Nodelet service running on the nodes will keep checking with the sunpike for any such custom configuration and if found, will copy the custom configuration on the kubernetes host(master/worker) and restart the kubelet service to apply the change.

Example usage:

Bash

curl --request POST \
 --url 'https://platform9.com/qbert/v4/{projectId}/kubeletconfig' \
 --header 'X-Auth-Token: {X-Auth-Token}' \
 --data '[
  {
    "clusterId": "{string}",
    "config": [
      {
        "configName": "{string}",
        "configData": "{string}",
        "role": "{string}",
        "nodeIps": [
          "{array[string]...}"
        ],
        "applyAll": "{boolean}"
      }
    ]
  }
]'

API Parameter	Type	Description
clusterId	string	Id of the cluster to which the config belongs.
config	array[object]	Array of the kubelet configuration objects passed.

Kubelet Configuration Object Parameters	Type	Description
configName	string	Name of the configmap to be created as kubelet configuration
configData	string	Base64 encoded string of the kubelet configuration configmap.
role	string	Role of the nodes for which the configuration is to be applied. Applicable values are master or worker or both.
nodeIps	array[string]	List of IPs of the nodes for which the configuration is to be applied.
applyAll	boolean	Set to true if configuration is to be applied to all nodes

Feature usage considerations and notes:

• Required: [clusterId,configName, configData,role ] and one of [nodeIps,applyAll ]

• Ensure that only 1 matching configmap should be present for each role/node for each cluster.

• You can update the configurations at any time. Nodelet will reapply change during reconcile and restart the kubelet service. There might be a time delay due to this, for the changes to take effect.

• The default config maps are for reference only. Only one of each master & worker config map will be created on the sunpike host, regardless of number of clusters deployed using the DU.

• This feature is not backported to kubernetes version < 1.24. For previous versions, user can continue with the old way of configuring dynamic kubelet.

Editing Kubelet Configurations

Existing Kubelet configurations can be updated using the Qbert Put Kubelet Config API. The api lets you update the dynamic kubelet configurations for a specific cluster, marked via the clusterid.

Sample usage:

Bash

curl --request PUT \
 --url 'https://platform9.com/qbert/v4/{projectId}/kubeletconfig' \
 --header 'X-Auth-Token: {X-Auth-Token}' \
 --data '[
  {
    "clusterId": "{string}",
    "config": [
      {
        "configName": "{string}",
        "configData": "{string}",
        "role": "{string}",
        "nodeIps": [
          "{array[string]...}"
        ],
        "applyAll": "{boolean}"
      }
    ]
  }
]'

Deleting Kubelet Configurations

Existing Kubelet configurations can be deleted using the Qbert Delete Kubelet Config API. The api lets you delete the existing dynamic kubelet configurations present in sunpike. You have to pass the clusterId's along with the list of the names of the kubelet configurations to be deleted for that cluster.

Sample usage:

Bash

curl --request DELETE \
 --url 'https://platform9.com/qbert/v4/{projectId}/kubeletconfig' \
 --header 'X-Auth-Token: {X-Auth-Token}'
 --data '[
    {
            "clusterId": "{string}", 
            "config": [{array[string]...}]
    },
    {
            "clusterId": "{string}", 
            "config": [{array[string]...}]
    },
]