> For the complete documentation index, see [llms.txt](https://docs.platform9.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.platform9.com/managed-kubernetes/5.11/pmk-release-notes/release-notes.md). # Release Notes {% hint style="info" %} **PMK 5.11.2 patch is available now** Release Notes: {% endhint %} ## Release Summary The Platform9 Managed Kubernetes (PMK) version 5.11 release is now generally available with active support for Kubernetes **v1.30**. This release brings new features, enhancements and critical bug fixes to improve overall user experience and stability. {% hint style="danger" %} **Kubernetes 1.27 and lower EOL** Kubernetes v1.27 and lower versions are marked as End of Life on PMK 511. New clusters are to be created on Kubernetes v1.29 or above. All clusters must be upgraded to at least Kubernetes v1.28 before upgrading from PMK 5.10.x to PMK 5.11. {% endhint %} {% hint style="warning" %} **Kubernetes 1.28 Deprecated** Kubernetes v1.28 is marked as deprecated. New clusters should be created on Kubernetes v1.29 or above. {% endhint %} ## PMK 5.11.0 Release Highlights (Released 2024-10-16) #### New Features * Added active support to Kubernetes 1.30. * Added support for Rocky 9.4: [auto$](https://github.com/platform9/pcd-docs-gitbook/blob/main/kubernetes/support-matrix/README.md) * Added a new webpage for Custom Resource Definitions on the Kubernetes app under `cluster administration`. * You can now manage CRDs and CRs from the the PMK UI. #### Feature Updates * PMK monitoring helm chart is now upgraded to release v62.7.1. The component upgrade includes Grafana upgrade to v11. * You can now add a custom Grafana service URL to PMK UI. See: [auto$](https://github.com/platform9/pcd-docs-gitbook/blob/main/kubernetes/adding-custom-grafana-url-to-pmk-ui/README.md) * Updated the PMK monitoring helm chart to use upstream scrape config to use the upstream rules, reducing the overhead in maintaining the helm chart. See: [auto$](https://github.com/platform9/pcd-docs-gitbook/blob/main/kubernetes/pf9-kube-prometheus-helm-chart-upgrade/README.md) * Added Golang Security fixes in sunpike services. #### Deprecations, Feature Removal and EOL information * Following Operating Systems are no longer supported with PMK 5.11 and above. Check the PMK 5.11 support here: [auto$](https://github.com/platform9/pcd-docs-gitbook/blob/main/kubernetes/support-matrix/README.md) * RHEL 7.9 * RHEL 8.5 * Ubuntu 18 #### Platform9 CLI The *pf9ctl* release 1.29 is now available and can be installed by running the following command. {% tabs %} {% tab title="Bash" %} ```bash bash <(curl -sL https://pmkft-assets.s3-us-west-1.amazonaws.com/pf9ctl_setup) ``` {% endtab %} {% endtabs %} {% hint style="danger" %} **Mandatory AWS cluster changes** All existing and new AWS clusters in PMK must be configured with an is\_update flag and restricted security group rules. Without this cluster updates(such as AMI updates) and upgrades may fail. Please reach out to Platform9 support for this configuration. {% endhint %} ### Bug Fixes `Fixed`Fixed a bug that caused NO\_PROXY set through pf9ctl set-proxy not being set in `containerd 00-pf9-proxy.conf` file. `Fixed` Editing DNS Domain under the `Edit DNS add-on` page on UI is not allowed anymore. Updating the DNS Domain after cluster creation can lead to a cluster being non-functional. `Fixed` Large text in Kubernetes resource YAML search would earlier break the page and display a blank page instead. ### Known Issues `Known Issue` All existing and new AWS clusters in PMK must be configured with an is\_update flag and restricted security group rules. Without this cluster updates(such as AMI updated) and upgrades may fail. Please reach out to Platform9 support for this configuration. `Known Issue` During upgrade of a PMK cluster, uninstallation of pf9-kube package may be incomplete/ stuck, if there are any workloads whose associated containers cannot be cleanly stopped and removed. Contact platform9 support if this is observed. `Known Issue` On Rocky Linux 9 (tested on 9.2 and 9.4), platform9's pf9-kube package installs `iptables-services` as a dependency. With recent updates to the upstream repositories, installation will fail due to a missing dependency on `iptables-legacy-*` packages. `Known Issue` (On Rocky Linux 9:) Users will need to install the legacy packages by running `dnf install iptables` or `dnf install iptables-utils` on workload cluster nodes. Since this is a recent upstream change, a solution will be provided in upcoming releases by packaging the required packages along with `pf9-kube` package. `Known Issue` AWS clusters using flannel CNI need to be updated to use port 2379 instead of 4001 from1.22 version onwards. Workaround is to go to the "Edit cluster" option on the UI and clicked on "Update cluster" without making any changes. This adds the 2379 ingress rule to the master ELB. `Known Issue` When a detach operation is performed on a master node in a multi master cluster, it takes approximately 30 minutes to complete all the detach operations and perform cleanup on the node. Therefore, if you want to reattach this node to any other cluster, you need to wait for the nodelet to stop all the phases and perform cleanup before attempting to reattach the node. `Known Issue` In some scenarios, after a node is removed from the qbert clusters, nodelet fails to cleanup the data. Workaround is to check and remove the /var/opt/pf9/kube directory if present, even after the node is deauthorized. `Known Issue` Cluster upgrade attempt is blocked on UI post a cluster upgrade failure due to nodes being in a converging/not converged state. `Known Issue` Kubelet authorization mode is marked set to AlwaysAllow instead of Webhook. `Known Issue` PMK Cloud provider created directly in Sunpike cannot be used to create qbert clusters. Qbert cloud providers will work to create both qbert and sunpike clusters. But cloud providers created directly in sunpike CANNOT be used to create qbert clusters. Please use the appropriate one based on your needs. `Known Issue` Disabling Luigi Advanced Networking Operator (Luigi) add-on deletes cert-manager CRDs as well as part of dependencies cleanup. Please reach out to Platform9 for assistance on Luigi removal. ### Package Updates #### PMK 5.11.0 Latest Kubernetes Components List | Component | Kubernetes 1.30 | Kubernetes 1.29 | Kubernetes 1.28 | | ------------------------------------ | ---------------- | ------------------ | ------------------ | | KUBERNETES BUILD VERSION | 1.30.4-pmk.**2** | 1.29.2-pmk.**162** | 1.28.6-pmk.**185** | | CONTAINERD | 1.7.13 | 1.7.13 | 1.7.13 | | RUNC | 1.1.12 | 1.1.12 | 1.1.12 | | CORE-DNS | 1.11.1 | 1.11.1 | 1.11.1 | | METRICS SERVER | 0.6.4 | 0.6.4 | 0.6.4 | | METAL LB | 0.14.2 | 0.14.2 | 0.14.2 | | KUBERNETES DASHBOARD | 2.7.0 | 2.7.0 | 2.7.0 | | CLUSTER AUTO-SCALER AWS | 1.28.0 | 1.28.0 | 1.28.0 | | FLANNEL | 0.24.2 | 0.24.2 | 0.24.2 | | CALICO | 3.27.2 | 3.27.2 | 3.27.2 | | ETCD | 3.5.12 | 3.5.12 | 3.5.12 | | CNI PLUGINS | 1.4.0 | 1.4.0 | 1.4.0 | | KUBEVIRT | 1.0.0 | 1.0.0 | 1.0.0 | | KUBEVIRT CDI | 1.57.0 | 1.57.0 | 1.57.0 | | ADVANCED NETWORKING OPERATOR (LUIGI) | 0.5.6 | 0.5.6 | 0.5.6 | | MONITORING - PROMETHEUS OPERATOR | 0.68.1 | 0.68.1 | 0.68.1 | | PROFILE AGENT | 2.0.2 | 2.0.2 | 2.0.2 | | METAL3 | 1.1.1 | 1.1.1 | 1.1.1 | ## [PMK 5.11.1 Patch](https://app.developerhub.io/platform9-docs/5.11/kubernetes/release-notes#pmk511-patch-update) (Released 2025-01-31) This patch adds support for RHEL 8.10 and fixes the following issues with updated kube patches and a new release of pf9ctl CLI. There is no need to update the PMK management plane to a version 5.11.1 - the management plane upgrade to v5.11.0 is sufficient. `Added` Added support for RHEL 8.10 (Refer [Managed Kubernetes Support Matrix](https://platform9.com/docs/5.11/kubernetes/support-matrix)) `Added` Updated golang to v1.23 and updated libraries/ module dependencies to fix CVEs ### Bug Fixes `Fixed` \[PMK-6596] : kube-scheduler and kube-controller services exposed on all interfaces, risking external access : `Solution` Access to kube-apiserver, kube-scheduler and kube-controller services is restricted within the cluster and these services cannot be reached from outside the cluster `Fixed` \[PMK-6606] : Ability to turn debugging on/off was missing for SMCP management clusters. This has been fixed with a new Kube patch for k8s 1.30 (build 1.30.4-pmk.80). However, the fix will be available in SMCP with the upcoming patch `SMCP 5.11.1` `Solution`Added variable "allowDebug" for nodelet bootstrap config file (located at bootstrapCfgPath in airctl-config.yaml) which is used to create the management cluster. Add/update this variable before running airctl create-mgmt/upgrade-mgmt command. This variable will modify verbosity level cluster-wide(for all master as well as worker nodes). The default value is false. Setting it to true will set verbosity level of kube-apiserver, kube-controller-manager, kube-scheduler and kubelet to 8. Setting it to false will set verbosity level to 2 for the same. #### Platform9 CLI The *pf9ctl* release 1.30 is now available ([release notes for pf9ctl v1.30](https://github.com/platform9/pf9ctl/releases/tag/1.30)) and can be installed by running the following command {% tabs %} {% tab title="Bash" %} ```bash bash <(curl -sL https://pmkft-assets.s3-us-west-1.amazonaws.com/pf9ctl_setup) ``` {% endtab %} {% endtabs %} ### Known Issues **Same as PMK 5.11.0** (see above [known issues in PMK 5.11.0](https://platform9.com/docs/5.11/kubernetes/release-notes#known-issues)) {% hint style="warning" %} **Supported cluster versions when upgrading from PMK 5.10** When upgrading from PMK 5.10 (5.10.0 or patches), existing k8s clusters with v1.28 or v1.29 (supported in PMK 5.10 too) must be upgraded to the following kube builds and not earlier kube builds {% endhint %} #### PMK 5.11.1 Latest Kubernetes Components List | Component | Kubernetes 1.30 | Kubernetes 1.29 | Kubernetes 1.28 | | ------------------------------------ | ----------------- | ------------------ | ------------------ | | KUBERNETES BUILD VERSION | 1.30.4-pmk.**80** | 1.29.2-pmk.**230** | 1.28.6-pmk.**271** | | CONTAINERD | 1.7.13 | 1.7.13 | 1.7.13 | | RUNC | 1.1.12 | 1.1.12 | 1.1.12 | | CORE-DNS | 1.11.1 | 1.11.1 | 1.11.1 | | METRICS SERVER | 0.6.4 | 0.6.4 | 0.6.4 | | METAL LB | 0.14.2 | 0.14.2 | 0.14.2 | | KUBERNETES DASHBOARD | 2.7.0 | 2.7.0 | 2.7.0 | | CLUSTER AUTO-SCALER AWS | 1.28.0 | 1.28.0 | 1.28.0 | | FLANNEL | 0.24.2 | 0.24.2 | 0.24.2 | | CALICO | 3.27.2 | 3.27.2 | 3.27.2 | | ETCD | 3.5.12 | 3.5.12 | 3.5.12 | | CNI PLUGINS | 1.4.0 | 1.4.0 | 1.4.0 | | KUBEVIRT | 1.0.0 | 1.0.0 | 1.0.0 | | KUBEVIRT CDI | 1.57.0 | 1.57.0 | 1.57.0 | | ADVANCED NETWORKING OPERATOR (LUIGI) | 0.5.6 | 0.5.6 | 0.5.6 | | MONITORING - PROMETHEUS OPERATOR | 0.68.1 | 0.68.1 | 0.68.1 | | PROFILE AGENT | 2.0.2 | 2.0.2 | 2.0.2 | | METAL3 | 1.1.1 | 1.1.1 | 1.1.1 | ## [PMK 5.11.2 Patch](https://app.developerhub.io/platform9-docs/5.11/kubernetes/release-notes#pmk511-patch-update) (Released 2025-03-19) This patch comes with improved UI performance, updated kube patches and a new release of pf9ctl CLI. PMK management plane needs to be updated to version 5.11.2 `Added` UI performance improvements when loading a large number of Kubernetes objects like namespaces, pods, deployments, services etc. with improved caching strategy in rendering. `Added` Updated libraries/ module dependencies to fix CVEs `Added` Added ability to modify the verbosity level for kube-api server. ### Bug Fixes `Fixed` Fixed the issue causing MetalLB addon in BGP mode to report unhealthy state due to the bug [Community validation webhook denies all updates](https://github.com/metallb/metallb/issues/2597) in upstream MetalLB v0.14.2. Upgraded MetalLB addon to [v0.14.9](https://metallb.universe.tf/release-notes/#version-0-14-9) to solve the issue. `Fixed` Fixed an issue causing host onboarding to fail for new hosts on DU having dynamic kubelet configuration DKC. `Fixed` Fixed the issues that caused add-on operator to remove the custom cert manager objects on disabling Luigi and custom prometheus objects on disabling the pf9-monitoring add-on. `Fixed` Fixed an issue due to which existing default apiserver flags could not be modified (e.g. `service-account-issuer`). #### Platform9 CLI The *pf9ctl* release 1.31 is now available ([release notes for pf9ctl v1.31](https://github.com/platform9/pf9ctl/releases/tag/1.31)) and can be installed by running the following command {% tabs %} {% tab title="Bash" %} ```bash bash <(curl -sL https://pmkft-assets.s3-us-west-1.amazonaws.com/pf9ctl_setup) ``` {% endtab %} {% endtabs %} ### Known Issues **Same as PMK 5.11.0** (see above [known issues in PMK 5.11.0](https://platform9.com/docs/5.11/kubernetes/release-notes#known-issues)) {% hint style="warning" %} **Supported cluster versions when upgrading from PMK 5.10** When upgrading from PMK 5.10 (5.10.0 or patches), existing k8s clusters with v1.28 or v1.29 (supported in PMK 5.10 too) must be upgraded to the following kube builds and not earlier kube builds. {% endhint %} #### PMK 5.11.2 Latest Kubernetes Components List | Component | Kubernetes 1.30 | Kubernetes 1.29 | Kubernetes 1.28 | | ------------------------------------ | ------------------ | ------------------ | ------------------ | | KUBERNETES BUILD VERSION | 1.30.4-pmk.**139** | 1.29.2-pmk.**245** | 1.28.6-pmk.**287** | | CONTAINERD | 1.7.13 | 1.7.13 | 1.7.13 | | RUNC | 1.1.12 | 1.1.12 | 1.1.12 | | CORE-DNS | 1.11.1 | 1.11.1 | 1.11.1 | | METRICS SERVER | 0.6.4 | 0.6.4 | 0.6.4 | | METAL LB | 0.14.**9** | 0.14.**9** | 0.14.**9** | | KUBERNETES DASHBOARD | 2.7.0 | 2.7.0 | 2.7.0 | | CLUSTER AUTO-SCALER AWS | 1.28.0 | 1.28.0 | 1.28.0 | | FLANNEL | 0.24.2 | 0.24.2 | 0.24.2 | | CALICO | 3.27.2 | 3.27.2 | 3.27.2 | | ETCD | 3.5.12 | 3.5.12 | 3.5.12 | | CNI PLUGINS | 1.4.0 | 1.4.0 | 1.4.0 | | KUBEVIRT | 1.0.0 | 1.0.0 | 1.0.0 | | KUBEVIRT CDI | 1.57.0 | 1.57.0 | 1.57.0 | | ADVANCED NETWORKING OPERATOR (LUIGI) | 0.5.6 | 0.5.6 | 0.5.6 | | MONITORING - PROMETHEUS OPERATOR | 0.68.1 | 0.68.1 | 0.68.1 | | PROFILE AGENT | 2.0.2 | 2.0.2 | 2.0.2 | | METAL3 | 1.1.1 | 1.1.1 | 1.1.1 | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.platform9.com/managed-kubernetes/5.11/pmk-release-notes/release-notes.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.