etcd Secrets Encryption

As of Platform9 Managed Kubernetes v5.2, Kubernetes clusters running version 1.20 and later can be configured to encrypt secrets that are stored in etcd.

To setup a cluster to use Secrets Encryption, the encryption provider config YAML needs to be created on each master node prior to creating the cluster and must reside under /var/opt/pf9/kube/apiserver-config directory and the same absolute path must be provided when creating the cluster.

To enable secrets encryption add the following custom API Server Flag:

"apiServerFlags": "--encryption-provider-config=/var/opt/pf9/kube/apiserver-config/encryption-provider.yaml"

Advanced Feature

Please contact Platform9 Support for assistance in configuring clusters with secrets encryption.

Cluster creation

To create a cluster with encryption each Kubernetes master node must contain the encryption provider configuration prior to creating the cluster.

IMPORTANT

The file must be located under /var/opt/pf9/kube/apiserver-config directory. This is the only directory that will be mounted in the API server container.

To create the file and directory run the commands below.

Bash

mkdir -p /var/opt/pf9/kube/apiserver-config # This directory may be missing in case the node has not been preped using pf9ctl
touch /var/opt/pf9/kube/apiserver-config/encryption-provider.yaml
chown pf9:pf9group /var/opt/pf9/kube/apiserver-config/encryption-provider.yaml

Refer to Encrypting Secret Data at Rest for contents of this file. Replace the key name and secret with your valid configuration.

A sample config:

YAML

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: ZnfoLrhTJFRq3pl4cPrlboH0lsHS33A3axWDl1HKWj8=
    - identity: {}

Create a cluster with secret encryptions using the Qbert API

To create a cluster with secrets encryption using the Qbert API an additional field is required in the clusters POST command. The field "apiServerFlags"needs to be provided with the location of the encryption provider yaml.

YAML

URL: qbert/v4/<project ID>/clusters
Method: POST
Headers: X-Auth-Token, Content-Type
Body: 
{
	"name": <>,
	"allowWorkloadsOnMaster": true,
	"containersCidr": "10.20.0.0/22",
	"servicesCidr": "10.21.0.0/22",
	"mtuSize": 1440,
	"privileged": true,
	"appCatalogEnabled": false,
	"nodePoolUuid": <>,
	"kubeRoleVersion": "1.20.5-pmk.1774",
	"calicoIpIpMode": "Always",
	"calicoNatOutgoing": true,
	"calicoV4BlockSize": "26",
	"calicoIPv4DetectionMethod": "first-found",
	"networkPlugin": "calico",
	"runtimeConfig": "",
	"etcdBackup": {
		"storageType": "local",
		"isEtcdBackupEnabled": 1,
		"storageProperties": {
			"localPath": "/etc/pf9/etcd-backup"
		},
		"intervalInMins": 1440
	},
	"apiServerFlags": "--encryption-provider-config=/var/opt/pf9/kube/apiserver-config/encryption-provider.yaml"
}
Response:
200 OK
{
    "uuid": "885ba8fb-50cf-48a7-bb38-d973cece8abf"
}

Attach the master nodes using Qbert API

If you are using the API to create the cluster you will need to attach each master node using the Qbert API.

uuid: The UUID can be found on the Infrastructure dashboard Nodes tab by adding UUID to the table.

YAML

URL: /qbert/v3/<project ID>/clusters/<cluster UUID from step 2>/attach
Method: POST
Headers: X-Auth-Token, Content-Type
Body: 
[{
	"uuid": "d454dac1-7e1b-417d-af2f-367efc91cf89",
	"isMaster": true
}]
Response:
200 OK
OK

Rotating a decryption key

To rotate the keys currently in use the encryption-provider.yaml file needs to be updated with the new key on each master node and the Kubernetes services must be restarted.

To update the encryption key a new key must first be generated in your Key Management System (KMS) and then added to encryption-provider.yamlas the second key entry for the current provider on all master nodes.

Once the encryption-provider.yaml has been updated on each master node stop the Platform9 services on each master node. To avoid a cluster outage progress through each node one by one ensuring that the master node converges and returns into the cluster before moving to the next.

Bash

systemctl stop pf9-nodeletd pf9-hostagent

Once each node has been restarted the Kubernetes API Server must be restarted to ensure that all nodes can decrypt using the new key. This may cause an outage if all nodes are restarted simultaneously,

Platform9 starts the apiserver as a container, to stop the kube-apiserver you will need to restart the entire stack. (Work is in inprogress to enable light-touch restarts.)

Bash

/opt/pf9/nodelet/nodeletd phases stop

/opt/pf9/nodelet/nodeletd phases start

Once the Kubernetes API server has started edit encryption-provider.yaml to promote the new key to be the first entry in the keys array. Promoting the new key to the first position in the array ensures that it is used for encryption.

Once the new key is promoted the Kubernetes API Server will need to be restarted for a second time.

Restart all kube-apiserver processes to ensure each server now encrypts using the new key.

Bash

/opt/pf9/nodelet/nodeletd phases stop

/opt/pf9/nodelet/nodeletd phases start

Once the Kubernetes API Server has restarted runkubectl get secrets --all-namespaces -o json | kubectl replace -f - to encrypt all existing secrets with the new key.

To complete the process backup etcd and then remove the old decryption key from the config.

Decrypting all data

To disable encryption at rest move the identity provider entry to be read as the first entry in the config:

YAML

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - identity: {}
    - aescbc:
        keys:
        - name: key1
          secret: <BASE 64 ENCODED SECRET>

Once the encryption-provider.yaml has been updated restart all kube-apiserver processes.

Platform9 starts the apiserver as a container, to stop the kube-apiserver we need to restart the entire stack.

To force all the secrets to be decrypted run the command below.

Bash

kubectl get secrets --all-namespaces -o json | kubectl replace -f -