Release Notes

PMK 5.10.2 patch update is available now

Release notes : https://platform9.com/docs/5.10/kubernetes/release-notes#pmk-5102-patch-update

Release Summary

The Platform9 Managed Kubernetes (PMK) version 5.10 release is now generally available with active support for Kubernetes v1.29. This release brings new features, enhancements and critical bug fixes to improve overall user experience and stability.

Kubernetes 1.26 and lower EOL

Kubernetes v1.26 and lower versions are marked as End of Life on PMK 510. New clusters are to be created on Kubernetes v1.28 or above. All clusters must be upgraded to at least Kubernetes v1.27 before upgrading from PMK 5.9.x to PMK 5.10.

Kubernetes 1.27 Deprecated

Kubernetes v1.27 is marked as deprecated. New clusters should be created on Kubernetes v1.28 or above.

PMK 5.10.0 Release Highlights (Released 2024-06-04)

New Features

Added active support to Kubernetes 1.29.
Added support for RHEL 8.9: auto$
Added new parameters to Qbert Upgrade API to support configuration of Calico and API server flags. Please see https://platform9.com/docs/qbert/ref#postupgrade-a-cluster-identified-by-the-uuid for details.
- addonOperatorImageTag
- apiServerFlags
- allowWorkloadsOnMaster
- calicoControllerCpuLimit
- calicoControllerMemoryLimit
- calicoIPv4DetectionMethod
- calicoNodeCpuLimit
- calicoNodeMemoryLimit
- calicoTyphaCpuLimit
- calicoTyphaMemoryLimit:
- controllerManagerFlags
Added functionality to modify and persist PMK add-on resource Requests and Limits change. Read more here: auto$
MetalLB upgraded to 0.13.11 and configuration updated to Custom Resource Definitions in conformance with upstream change https://metallb.universe.tf/#backward-compatibility.
- All new clusters created via UI will be created with Custom Resources created on cluster for configuration.
- All new cluster from Qbert API should provide MetalLB in form of base64 encoded Custom Resource yaml via the new parameter added to the Qbert API called base64EncMetallbConfig. Older parameter metallbCid should no longer recommended be used and will be deprecated in future versions. Read more here: https://platform9.com/docs/kubernetes/metallb-addon2#qbert-api-support
- Existing clusters with older config-map type configuration will continue to work. It is recommended to migrated to the newer configuration. Read https://metallb.universe.tf/configuration/migration_to_crds/ and MetalLB Application Load Balancer Addon for migration steps.
Improved clusters view.

Added new options to left navigations in UI.
- Users can now manage Configurations such as Resource Quotas, Limit Ranges, Horizontal Pod Autoscalers and Pod Disruption Budgets from UI.
- Users can now manage Networking configurations such as Endpoints, Ingresses and NetworkPolicies from UI.

Deprecations, Feature Removal and EOL information

Following types of clusters are removed from PMK 5.10 and will no longer be supported.
- Imported clusters - AKS, EKS, & GKE
- Azure Native clusters.
- CAPI clusters.
- One Click clusters

Platform9 CLI

The 1.27 pf9ctl release is now available and can be installed by running the following command.

bash <(curl -sL https://pmkft-assets.s3-us-west-1.amazonaws.com/pf9ctl_setup)

Added an option in PF9Cli to move an existing node behind proxy.

Bug Fixes

Fixed Fixed an issue due to which certificate generation fails if CA validity is less than the amount of TTL with which the certificate is attempted to be generated in Vault

Fixed Fixed an issue due to which services were not getting IPs from the updated metalLB address pool.

Fixed Fixed an issue due to which Calico pods went into crashloopbackoff state on worker nodes on AWS clusters.

Fixed Fixed an issue which caused the Etcd container to log to a file not accessible without Sudo access.

Known Issues

Known Issue On rebooting worker node, the kube stack may take more than 15 minutes to come into a healthy state, due to kubelet service being down and being restarted by nodelet.

Known Issue In the case when a node used with proxy is re-used, the Proxy configuration files get left over even after node decommission, not allowing the node to be re-used in another cluster. Workaround is to remove the file at the location: /etc/systemd/system/containerd.service.d/00-pf9-proxy.conf

Known Issue AWS clusters using flannel CNI need to be updated to use port 2379 instead of 4001 from1.22 version onwards. Workaround is to go to the “Edit cluster” option on the UI and clicked on “Update cluster” without making any changes. This adds the 2379 ingress rule to the master ELB.

Known IssueWhen a detach operation is performed on a master node in a multi master cluster, it takes approximately 30 minutes to complete all the detach operations and perform cleanup on the node. Therefore, if you want to reattach this node to any other cluster, you need to wait for the nodelet to stop all the phases and perform cleanup before attempting to reattach the node.

Known Issue In some scenarios, after a node is removed from the qbert clusters, nodelet fails to cleanup the data. Workaround is to check and remove the /var/opt/pf9/kube directory if present, even after the node is deauthorized.

Known Issue Cluster upgrade attempt is blocked on UI post a cluster upgrade failure due to nodes being in a converging/not converged state.

Known Issue Kubelet authorization mode is marked set to AlwaysAllow instead of Webhook.

Known Issue UI throws error when using SSO with Azure AD and passwordless logins.

Known Issue PMK Cloud provider created directly in Sunpike cannot be used to create qbert clusters. Qbert cloud providers will work to create both qbert and sunpike clusters. But cloud providers created directly in sunpike CANNOT be used to create qbert clusters. Please use the appropriate one based on your needs.

Known Issue Certificate generation fails if CA validity is less than the amount of TTL with which certificate is attempted to be generated in Vault

Package Updates

PMK 5.10.0 Latest Kubernetes Components List

Component

Kubernetes 1.29

Kubernetes 1.28

Kubernetes 1.27

KUBERNETES BUILD VERSION

1.29.2-pmk.82

1.28.6-pmk.120

1.27.13-pmk.205

CONTAINERD

1.7.13

RUNC

1.1.12

CORE-DNS

1.11.1

METRICS SERVER

0.6.4

METAL LB

0.13.11

KUBERNETES DASHBOARD

2.7.0

CLUSTER AUTO-SCALER AWS

1.28.0

FLANNEL

0.22.2

CALICO

3.27.2

3.26.3

ETCD

3.5.12

CNI PLUGINS

1.4.0

1.3.0

KUBEVIRT

1.0.0

KUBEVIRT CDI

1.57.0

ADVANCED NETWORKING OPERATOR (LUIGI)

0.5.5

MONITORING - PROMETHEUS OPERATOR

0.68.1

PROFILE AGENT

2.0.2

METAL3

1.1.1

PMK 5.10.2 Patch Update (Released 2024-09-13)

Patch release versioning

PMK 5.10.2 is the first generally available patch in the PMK 5.10.x series. PMK 5.10.1 was a hot-fix patch used for select use-cases only.

Mandatory AWS cluster changes

All existing and new AWS clusters in PMK must be configured with an is_update flag and restricted security group rules. Without this cluster updates(such as AMI updates) and upgrades may fail. Please reach out to Platform9 support for this configuration.

Added Added the ability to customize the hostplumber metrics port. The port can be set by setting metricsPort in network plugins configuration. See sample configuration.

Added Added a flag named skip-os-check in pf9ctl to bypass the supported OS check during prep-node command execution.

Added Added the functionality to specify the location for the containerd storage on the nodes. Read more: auto$

Added Added support for Dynamic Kubelet Configuration via Qbert API. Read more: auto$

Fixed Fixed a bug that caused a nodelet phases restart on nodes due to connection interruption between the node and the kube-apiserver. This led to unintended node drain and workload downtime.

Fixed MetalLB is upgraded to v0.14.2 which fixes the following known issue (resolved in MetalLB 0.14.2 upstream) - If there were one or more LB services with external IP in pending state, and if controller restarts, the external IP may be reassigned/shuffled between the existing services. This was due to MetalLB controller design, on restarts, it first looked at pending services and assigned them the IP from pool without checking if the IP was already assigned to another service.

Fixed On rebooting a worker node, the kube stack and the node would take up to 15 minutes to come to a healthy and ready state.

Fixed Added secure ciphers for Luigi and Addon-operator

Known Issue All existing and new AWS clusters in PMK must be configured with an is_update flag and restricted security group rules. Without this cluster updates(such as AMI updated) and upgrades may fail. Please reach out to Platform9 support for this configuration.

Known Issue During upgrade of a PMK cluster, uninstallation of pf9-kube package may be incomplete/ stuck, if there are any workloads whose associated containers cannot be cleanly stopped and removed. Contact platform9 support if this is observed.

Process Change Support Bundle generation and upload process is updated now with following changes:

Starting PMK 5.10.2 and above versions, the generated support bundle will be gpg encrypted. The encryption key from https://gpg.platform9.com/publickey.txt is sourced and placed at /etc/pf9/public_key.asc on hosts.
Users are required to upload only the encrypted support bundles to Platform9 support.
Generated support bundles are now redacted of any sensitive data such as certificates, token, passwords etc.
The encrypted support bundle file name contains fingerprint of the public key used to encrypt the bundle using the following pattern: /tmp/pf9-support.tgz.<GPG key fingerprint>.gpg
pf9ctl does not support support bundle generation and auto-uploads starting PMK 5.10.2.
Procedure to generate a support bundle using the datagatherer script can be found here.
Please reach out to Platform9 support to get on-boarded to the new the upload process.

PMK 5.10.2 Latest Kubernetes Components List