# Troubleshooting host onboarding issues

This document describes how to identify and resolve issues that occur when onboarding a host using the `pcdctl prep-node` command.

## Most Common Causes

* Residual configuration from a prior failed node preparation can cause role assignment or installation to fail.
* The user running the prep-node command may not have proper `sudo` privileges, causing command failures.
* Packages are corrupted or not installed properly; `dpkg` or `apt` lock files preventing package installation or updates during prep-node execution. Review `/var/log/dpkg.log` to verify whether packages are partially installed or misconfigured.
* Incorrect proxy configurations provided to `prep-node` can block the download of required packages or scripts.
* `Firewalld` or other firewall rules may block required ports, preventing communication with the management plane.
* NTP is not synchronized, which may cause authentication or communication failures.
* Connectivity to the <code class="expression">space.vars.product\_name</code> management plane controller is broken or unreachable.

## Steps To Troubleshoot

The `pcdctl prep-node` without a prior configuration prompts interactively for account information that can be directly retrieved from the **GUI -> Infrastructure -> Cluster Hosts -> Add a New Host**. The GUI is pre-filled with the Account URL, Username, Region, and Tenant for the logged-in user.

All the configuration details like the Platform9 Account URL, Username, Password, Region, and Tenant are persisted in a local `config.json` file under `/pf9/db/`.

Logs for `pcdctl` command execution are stored in the `/pf9/logs/pcdctl-<DATE>.log` file.

{% stepper %}
{% step %}

### Review pcdctl logs

Start by reviewing the pcdctl log file to trace the exact error.

{% tabs %}
{% tab title="pcdctl logs" %}

```bash
"msg":"Received a call to fetch keystone authentication for fqdn: https://[FQDN] and user: [USER] and tenant: [TENANT], mfa_token: <br>"}
"msg":"Error calling keystone API:Post \"https://[FQDN]/keystone/v3/auth/tokens?nocatalog\": dial tcp: lookup example1.pcd.platform9.co on 127.0.0.53:53: no such host<br>"}
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

### Verify input values

Ensure to provide all the details correctly, without typos and extra space. The installer authenticates with the Keystone API using these values. Any incorrect entry can cause authentication failure or DNS resolution errors.
{% endstep %}

{% step %}

### Verify network connectivity

Verify that the host has outbound network connectivity to the internet and the <code class="expression">space.vars.product\_name</code> management plane controller:

* `$ curl -s https://<FQDN>`
* `$ ping www.google.com`
* `$ telnet`[`www.google.com`](http://www.google.com/)`443`
  {% endstep %}

{% step %}

### Gather verbose logs

Execute `pcdctl prep-node` with the `--verbose` flag to gather detailed logs of the host preparation process, including each command executed, checks performed, and any warnings or errors encountered.
{% endstep %}

{% step %}

### Validate prerequisites and host checks

Ensure the [primary prerequisites](https://platform9.com/docs/private-cloud-director/private-cloud-director/pre-requisites#hypervisor-host-prerequisites) are met. Review the checks below for additional validation.

* Verify the host is running a supported Ubuntu version. Currently, Platform9 supports Ubuntu 22.04 and 24.04 for Private Cloud Director host onboarding.

{% tabs %}
{% tab title="Bash" %}

```bash
$ sudo cat /etc/os-release | grep -E '^NAME=|^VERSION_ID='
```

{% endtab %}
{% endtabs %}

* Confirm the host has sufficient CPU cores and memory. Minimum 8 CPU cores and 16 GB RAM are recommended for host onboarding.

{% tabs %}
{% tab title="Bash" %}

```bash
# CPU cores
$ sudo grep -c ^processor /proc/cpuinfo

# Total memory
$ sudo free -h | grep Mem:
```

{% endtab %}
{% endtabs %}

* Verify the root partition (`/`) has adequate free space. Minimum 250 GB of free disk space is required.

{% tabs %}
{% tab title="Bash" %}

```bash
$ sudo df -h /
```

{% endtab %}
{% endtabs %}

* Ensure no other package manager (e.g., apt or dpkg) is running in the background. If either command returns a running process, wait for it to finish or terminate it before continuing.

{% tabs %}
{% tab title="Bash" %}

```bash
# Review if any dpkg, apt process is held
$ sudo lsof /var/lib/dpkg/lock
$ sudo lsof /var/lib/apt/lists/lock

# Review package manager logs
$ sudo cat /var/log/dpkg.log 
$ sudo cat /var/log/apt/history.log
```

{% endtab %}
{% endtabs %}

* Confirm the root or current user has passwordless sudo privileges; the user must have unrestricted sudo privileges for all operations.

{% tabs %}
{% tab title="Bash" %}

```bash
$ sudo -l
    (ALL : ALL) ALL
    (ALL) NOPASSWD: ALL
```

{% endtab %}
{% endtabs %}

* Check the status of `firewalld` to ensure it does not block Platform9 service communication. Platform9 recommends stopping and disabling `firewalld` on these hosts using `sudo systemctl stop firewalld` and `sudo systemctl disable firewalld`.

{% tabs %}
{% tab title="Bash" %}

```bash
$ sudo systemctl is-active firewalld
inactive
```

{% endtab %}
{% endtabs %}

* Ensure NTP is enabled for accurate time synchronization across hosts. Verify if `systemd-timesyncd` is already running on the host, as it provides basic time synchronization.

{% tabs %}
{% tab title="Bash" %}

```bash
$ sudo systemctl status systemd-timesyncd
$ sudo timedatectl status
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

### Proxy configuration (if applicable)

For [hosts using a proxy server for outbound connectivity](https://platform9.com/docs/private-cloud-director/2025.8/private-cloud-director/pre-requisites#connectivity-via-https-proxy-server), ensure that the `/etc/environment` file has required variables configured. Also configure the package manager (apt) to properly fetch required packages through the proxy server; refer to the linked documentation.
{% endstep %}

{% step %}

### Verify hostagent installation and logs

As the last step, the hostagent package is downloaded and installed on the host. Verify the `pf9-hostagent.service` status and `/var/log/pf9/hostagent.log` file to track the progress.

{% tabs %}
{% tab title="Bash" %}

```bash
$ sudo service pf9-hostagent status
$ sudo cat /var/log/pf9/hostagent.log
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

### Post-onboarding: authorization and role assignment

After onboarding the host to the <code class="expression">space.vars.product\_name</code>, it can be [Authorized & Assigned Roles](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#step-2-authorize-host-and-assign-roles). This involves downloading and installation of service specific packages and service initialization, which can be monitored through `/var/log/pf9/hostagent.log`. The `.deb` packages are downloaded inside `/var/cache/pf9apps` directory.
{% endstep %}

{% step %}

### Contact support

If these steps prove insufficient to resolve the issue, reach out to the [Platform9 Support team](https://support.platform9.com/) for additional assistance.
{% endstep %}
{% endstepper %}