Diagnose a Host Agent Stuck in Converging State

Overview

When a host is authorized and roles are assigned, PCD downloads and applies service packages through the host agent. During this process, the host shows a converging status in the UI. Under normal conditions, convergence completes within a few minutes. If a host remains in converging state for more than 10–15 minutes, the host agent has likely encountered an error it cannot recover from on its own.

In this guide, you will locate the relevant log files, identify the most common causes of a stuck convergence, and restore the host to applied status.

Check the Hostagent Log First

The hostagent log is the primary source of information when a host is stuck converging. SSH to the affected host and inspect the log:

tail -100 /var/log/pf9/hostagent.log

Or follow it live while monitoring:

tail -f /var/log/pf9/hostagent.log

Look for lines containing ERROR, WARN, failed, timeout, or permission denied. The section below maps the most common messages to their fixes.

Common Causes and Fixes

Service Start Failure

Symptom in log: Lines mentioning a specific pf9-* service (pf9-ostackhost, pf9-neutron-ovn-controller, etc.) failing to start or timing out.

The host agent applies roles by starting Platform9 services. If a service fails to start, the agent retries but eventually marks the convergence as failed.

What to check:

# Replace pf9-ostackhost with the service named in the log
sudo systemctl status pf9-ostackhost
sudo journalctl -u pf9-ostackhost --since "30 minutes ago" | tail -80

Common sub-causes:

• Port conflict: Another process is already bound to a port the service needs. Check with sudo ss -tlnp.

• Missing dependency: A required package was not installed. Check /var/log/dpkg.log and /var/log/apt/history.log for installation failures.

• Configuration error written by a prior failed attempt: Remove the stale role-state files and retry. See Re-Onboarding Recovery.

After addressing the root cause, restart the host agent to trigger re-convergence:

sudo systemctl restart pf9-hostagent

Log-Rotation Permission Error Filling Up the Agent

Symptom in log: Messages about failing to write to /var/log/pf9/hostagent.log, or the log file stops updating while the host remains in converging state in the UI.

Symptom on host:

df -h /var/log

If /var/log is at or near 100%, the hostagent cannot write logs and may be stuck waiting for I/O.

Fix:

1. Free up log space:

sudo journalctl --vacuum-size=1G
sudo find /var/log/pf9 -name "*.log.*" -mtime +7 -delete

1. Check and fix log file permissions. The hostagent log must be writable by the pf9 user or root:

ls -la /var/log/pf9/hostagent.log
sudo chown root:root /var/log/pf9/hostagent.log
sudo chmod 644 /var/log/pf9/hostagent.log

1. Restart the hostagent:

sudo systemctl restart pf9-hostagent

1. Monitor the log to confirm convergence resumes:

tail -f /var/log/pf9/hostagent.log

Package Download or Installation Failure

Symptom in log: References to pf9apps, apt-get, dpkg, or HTTP errors when fetching packages from the management plane.

The hostagent downloads .deb packages into /var/cache/pf9apps/ and installs them during convergence. Download failures cause the agent to retry until it gives up.

What to check:

# Confirm the management plane is reachable from the host
curl -sk https://<management-plane-fqdn>/resmgr/v1/hosts | head -c 200

# Check for stale or corrupted package files
ls -la /var/cache/pf9apps/

Fix:

1. Clear the package cache and let the agent re-download:

sudo rm -rf /var/cache/pf9apps/*
sudo systemctl restart pf9-hostagent

1. If the management plane is unreachable, resolve the network or proxy issue first. See Troubleshooting Offline or Failed Hosts.

Hostagent Crash Loop

Symptom: pf9-hostagent service shows failed or repeatedly restarts in systemctl status pf9-hostagent.

What to check:

sudo systemctl status pf9-hostagent
sudo journalctl -u pf9-hostagent --since "1 hour ago" | tail -100

Look for a panic or fatal error message. If the agent is crashing on startup, there may be a corrupted state file.

Fix:

# Stop the agent
sudo systemctl stop pf9-hostagent

# Remove state files that may be corrupted
sudo rm -f /opt/pf9/data/state/*

# Restart
sudo systemctl start pf9-hostagent

Monitor the log to verify the agent starts cleanly.

Trigger a Re-Sync from the UI

If the hostagent log shows no ongoing activity and the host remains in converging state, trigger a re-sync from the UI:

1. Navigate to Infrastructure > Cluster Hosts in the Private Cloud Director UI.

2. Select the affected host.

3. Click Other > Re-sync Host (if available).

This signals the management plane to re-send the desired role configuration to the host agent.

Alternatively, simply restarting the hostagent on the host achieves the same effect:

sudo systemctl restart pf9-hostagent

When to Contact Support

If the host does not reach applied status within 15 minutes of the restart, collect the following and contact Platform9 Support:

• The full hostagent log: /var/log/pf9/hostagent.log

• The output of sudo systemctl status pf9-hostagent

• The output of sudo journalctl -u pf9-hostagent --since "2 hours ago"

• For Self-Hosted deployments only, the output of airctl host-status

Self-Hosted deployments only. You can also check whether the management plane has a pending convergence task for the host:

airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml

Look for hosts where Status is not ok. In SaaS deployments, contact Platform9 Support if host-level steps do not resolve the convergence failure.