Troubleshooting host onboarding issues

This document describes how to identify and resolve issues that occur when onboarding a host using the pcdctl prep-node command.

Most Common Causes

• Residual configuration from a prior failed node preparation can cause role assignment or installation to fail.

• The user running the prep-node command may not have proper sudo privileges, causing command failures.

• Packages are corrupted or not installed properly; dpkg or apt lock files preventing package installation or updates during prep-node execution. Review /var/log/dpkg.log to verify whether packages are partially installed or misconfigured.

• Incorrect proxy configurations provided to prep-node can block the download of required packages or scripts.

• Firewalld or other firewall rules may block required ports, preventing communication with the management plane.

• NTP is not synchronized, which may cause authentication or communication failures.

• Connectivity to the Private Cloud Director management plane controller is broken or unreachable.

Steps To Troubleshoot

The pcdctl prep-node without a prior configuration prompts interactively for account information that can be directly retrieved from the GUI -> Infrastructure -> Cluster Hosts -> Add a New Host. The GUI is pre-filled with the Account URL, Username, Region, and Tenant for the logged-in user.

All the configuration details like the Platform9 Account URL, Username, Password, Region, and Tenant are persisted in a local config.json file under /pf9/db/.

Logs for pcdctl command execution are stored in the /pf9/logs/pcdctl-<DATE>.log file.

1

Review pcdctl logs

Start by reviewing the pcdctl log file to trace the exact error.

pcdctl logs

"msg":"Received a call to fetch keystone authentication for fqdn: https://[FQDN] and user: [USER] and tenant: [TENANT], mfa_token: <br>"}
"msg":"Error calling keystone API:Post \"https://[FQDN]/keystone/v3/auth/tokens?nocatalog\": dial tcp: lookup example1.pcd.platform9.co on 127.0.0.53:53: no such host<br>"}

2

Verify input values

Ensure to provide all the details correctly, without typos and extra space. The installer authenticates with the Keystone API using these values. Any incorrect entry can cause authentication failure or DNS resolution errors.

3

Verify network connectivity

Verify that the host has outbound network connectivity to the internet and the Private Cloud Director management plane controller:

• $ curl -s https://<FQDN>

• $ ping www.google.com

• $ telnetwww.google.com443

4

Gather verbose logs

Execute pcdctl prep-node with the --verbose flag to gather detailed logs of the host preparation process, including each command executed, checks performed, and any warnings or errors encountered.

5

Validate prerequisites and host checks

Ensure the primary prerequisites are met. Review the checks below for additional validation.

• Verify the host is running a supported Ubuntu version. Currently, Platform9 supports Ubuntu 22.04 and 24.04 for Private Cloud Director host onboarding.

Bash

$ sudo cat /etc/os-release | grep -E '^NAME=|^VERSION_ID='

• Confirm the host has sufficient CPU cores and memory. Minimum 8 CPU cores and 16 GB RAM are recommended for host onboarding.

Bash

# CPU cores
$ sudo grep -c ^processor /proc/cpuinfo

# Total memory
$ sudo free -h | grep Mem:

• Verify the root partition (/) has adequate free space. Minimum 250 GB of free disk space is required.

Bash

$ sudo df -h /

• Ensure no other package manager (e.g., apt or dpkg) is running in the background. If either command returns a running process, wait for it to finish or terminate it before continuing.

Bash

# Review if any dpkg, apt process is held
$ sudo lsof /var/lib/dpkg/lock
$ sudo lsof /var/lib/apt/lists/lock

# Review package manager logs
$ sudo cat /var/log/dpkg.log 
$ sudo cat /var/log/apt/history.log

• Confirm the root or current user has passwordless sudo privileges; the user must have unrestricted sudo privileges for all operations.

Bash

$ sudo -l
    (ALL : ALL) ALL
    (ALL) NOPASSWD: ALL

• Check the status of firewalld to ensure it does not block Platform9 service communication. Platform9 recommends stopping and disabling firewalld on these hosts using sudo systemctl stop firewalld and sudo systemctl disable firewalld.

Bash

$ sudo systemctl is-active firewalld
inactive

• Ensure NTP is enabled for accurate time synchronization across hosts. Verify if systemd-timesyncd is already running on the host, as it provides basic time synchronization.

Bash

$ sudo systemctl status systemd-timesyncd
$ sudo timedatectl status

6

Proxy configuration (if applicable)

For hosts using a proxy server for outbound connectivity, ensure that the /etc/environment file has required variables configured. Also configure the package manager (apt) to properly fetch required packages through the proxy server; refer to the linked documentation.

7

Verify hostagent installation and logs

As the last step, the hostagent package is downloaded and installed on the host. Verify the pf9-hostagent.service status and /var/log/pf9/hostagent.log file to track the progress.

Bash

$ sudo service pf9-hostagent status
$ sudo cat /var/log/pf9/hostagent.log

8

Post-onboarding: authorization and role assignment

After onboarding the host to the Private Cloud Director, it can be Authorized & Assigned Roles. This involves downloading and installation of service specific packages and service initialization, which can be monitored through /var/log/pf9/hostagent.log. The .deb packages are downloaded inside /var/cache/pf9apps directory.

9

Contact support

If these steps prove insufficient to resolve the issue, reach out to the Platform9 Support team for additional assistance.