Ctrlk

Common Issues

The purpose of this page is to provide solutions to common issues during Community Edition (CE) installation & operation.

Community Edition is not for production workloads. Use it for labs, evaluation, and learning only.

Resolving failed prerequisites checks

Ports should not be in use

The CE installer expects to be installed in a dedicated system with the following ports available: 443, 2379, 2380, 3306, 4194, 5395, 5672, 5673, 6264, 8023, 8158, 8285, 8558, 9080, 10250, 10255. Typically, this error will be encountered after a previous installation failure. To completely clean up an installation failure, use airctl delete-cluster and then retry the installation.

# Deletes the CE installation, including k3s
sudo airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml
# deletes logs & downloaded artifacts in the user's home directory
rm -r "$HOME/airctl-logs"
rm -r "$HOME/pcd-ce"
# runs the install script from the internet
curl -sfL https://go.pcd.run | bash

Insufficient CPU or RAM

The CE install script checks the number of logical CPUs available at system start. If the install fails this check, run the installation again after provisioning more logical CPUs to the system (if running as a virtual machine) or on physical hardware with more logical CPUs available.

The install script calculates the amount of usable memory using the MemTotal line in /proc/meminfo , and checks for an amount less than 28GB. Additional memory must be made available to the CE system.

Multiple default routes with the same metric

The CE install script checks for more than one default IPv4 route with the same metric. If more than one route with the same metric is found, this check will fail, as the system will not know which default gateway to use. To resolve this error, either change the metric on the additional default routes, or delete any additional default routes with the same metric if they aren't needed.

Unsupported OS or version

The CE install script checks the operating system name and version. If this check fails, please try installing using a supported operating system and version as described in the CE Pre-requisites.

Kernel panic or vm panic on oom

The CE install script checks to make sure kernel & oom (out of memory) panics are configured correctly. /proc/sys/kernel/panic should be set to 10 and /proc/sys/vm/panic_on_oom should be set to 0 .

Failed to check firewalld status or service not found

The CE install script checks systemctl to see if firewalld.service is inactive or not running, and will fail if the service is set to active. To resolve this, use the following commands.

IPv6 is disabled

The CE install script checks /proc/sys/net/ipv6/conf/all/disable_ipv6 to see if it is set to 0 . IPv6 should be enabled by default, but if it has been explicitly disabled, please enable IPv6 and reboot the CE system before trying the installation again.

Resolving installation failures

Calico CNI

CE installation can fail if it encounters issues with Calico CNI (container networking interface). Calico sets up networking within CE's Kubernetes pods and provides DNS resolution. Typically, these issues are either related to slowness when retrieving the container images from a remote image repository causing the install script to timeout while waiting, or are related to a previous installation not being fully cleaned up before an installation is attempted again. These issues can usually be resolved by fully cleaning up the installation attempt and trying again.

You may also consider setting the install's default timeout to an amount higher than 600 seconds. See the Extending default deployment timeout section of the Custom Installation page.

Failed to apply logrotation on node - exit status 127

The CE installation will fail if it is unable to create a cronjob to automatically rotate log files, and will return exit status 127 if cron is not installed on the system. This is a default package on the Ubuntu 22.04 AMD64 server cloud image distribution, but is not included with the Ubuntu 22.04 AMD64 minimal cloud image. Installing cron will resolve this issue, but using the minimal cloud image is not supported and doing so may result in additional install failures.

Curl responds with an error, or does not work

This is typically caused by network restrictions such as firewalls or lack of internet access. Using curl with verbose mode enabled (-v ) is useful for debugging.

Installation fails due to curl timeouts

This issue typically occurs early in the installation process, and can happen when the CIDRs used for Kubernetes services & pods conflicts with external local area networks. The default for the services CIDR is 10.21.0.0/16, and the default for the pods CIDR is 10.20.0.0/16. See the Community Edition deployment networking section on the Custom Installation page for information on how to change these defaults using environment variables. After the environment variables are set, use the same terminal session to follow the steps in the section titled "Recovering a failed installation" on this page.

Failed to find running pod percona-db-pxc-db-pxc-0

This error can occur when installing Community Edition on CPUs older than Intel Nehalem or AMD Bulldozer generations, as Percona requires the x86-64-v2 microarchitecture level. This can be confirmed by running the following command:

Look for Subdirectories of glibc-hwcaps directories in the output. The x86-64-v2 line should not say supported, searched .

Please try installing Community Edition again on CPUs newer than Intel Nehalem or AMD Bulldozer generations.

Required ports & external URLs

Required ports:

  • TCP: 53, 443

  • UDP: 53, 123

Should you need to allow access to external URLs in order to complete a Community Edition installation, here are all of the URLs that are currently accessed during install:

Here is the list of required external URLs in order to onboard a hypervisor host to Community Edition.

Recovering a failed installation

Situation: A Community Edition installation has failed, the issue has been rectified, and you'd like to restart the installation.

Resolution: You must first uninstall/delete the CE installation (and k3s) using the following command, and then run the install script again.

After these steps have completed successfully, you may complete the rest of the deployment process starting with the Local DNS Entries section in Getting Started With Community Edition.

General troubleshooting

For CE hosts, use the following commands to understand where the error is occurring:

  • Check CE's status after installing to see which region (Kubernetes namespace) is experiencing issues. /opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml

  • For troubleshooting a failed install, check the install logs at airctl-logs/airctl.log

  • Check the size and amount of available filesystem space df -h /

  • kubectl describe node will show information about CE's underlying Kubernetes infrastructure

    • if resources are constrained, the allocated resources block should show requests CPU and memory near or at 100%

  • kubectl get pods -A Look for any pods that are not Running or Completed. Pods in CrashLoopBackOff or with high restart counts and a young age should have their logs investigated for more detail.

  • kubectl logs <pod> -n <namespace> to view pod logs.

  • The default Kubernetes namespaces are pcd for the infrastructure region and pcd-community for the workload region, unless DU_FQDN and/or the REGION_NAME environment variables were customized prior to installation as described in the Deployment URL & region name section of Custom Installation.

PreviousCustom InstallationNextBackup and Restore CE

Last updated

Was this helpful?