# Platform9 Private Cloud Director Documentation
Complete technical documentation for Platform9 Private Cloud Director - the enterprise VMware alternative. Deploy and manage your private cloud with comprehensive guides, API references.
Platform9 Private Cloud Director
Learn about Platform9 products. Turn your existing infrastructure into a full-featured private cloud with Platform9. Manage VMs and containers at scale with a familiar user experience and automated APIs in a private, secure environment.
Cover image
Getting Started
Transition from VMware in minutes. Deploy VMs, configure networking, and manage storage with built-in automation, high availability, and unified control.
Enterprise private cloud, simplified. Deploy on bare metal or a VM with a single command, with full functionality, and experience a production-grade private cloud with zero licensing costs.
# Overview
Platform9's space.vars.product\_name is an enterprise private cloud platform that enables IT & DevOps administrator to manage their virtualized and Kubernetes infrastructure at scale while leveraging their existing investments in storage and server hardware.
space.vars.product\_name is built using an **open source KVM hypervisor** behind the scenes. It uses a combination of best-of-breed open source technologies for virtualization and container management, along with our unique features to provide a familiar experience to virtualization and container administrators, along with all enterprise capabilities required to build, maintain, and scale private clouds.
### Who is this documentation for?
The space.vars.product\_name documentation is for **administrators, operators** **and users** of space.vars.product\_name who are responsible for creating and maintaining virtualized clusters and applications provisioned on them, on behalf of their organization.
This documentation assumes a level of understanding of fundamental infrastructure management technologies such as virtualization, container orchestration, storage, networking etc.
### Benefit of space.vars.product\_namespace.vars.product\_name is a private cloud platform that enables you to manage your VMs and containers side by side. space.vars.product\_acronym offers following benefits to system administrators:
* **Familiar, virtualization first management interface** - space.vars.product\_name provides a familiar interface for virtualization administrators, where they can find all traditional virtualization features they've relied on.
* **All enterprise virtualization features built-in** - space.vars.product\_name comes built in with all key virtualization features such as VM HA, Clusters enabled with Distributed Resource Rebalancing & Scheduling, VM cloning and snapshotting, VM affinity, anti-affinity, and much more.
* **Integrates with all enterprise storage** - space.vars.product\_name integrates with a wide array of enterprise storage arrays and systems, enabling you to keep your existing storage investments. space.vars.product\_acronym also integrates with all your existing server hardware, without requiring a change or refresh.
* **In-place vSphere cluster conversion with vJailbreak** - In addition to space.vars.product\_name, Platform9 also offers a free migration tool for VMware users, called [vJailbreak](https://platform9.com/vjailbreak/). With vJailbreak, you can perform in-place conversion of your vSphere clusters in a rolling manner to convert them to space.vars.product\_name cluster, using automation.
### Design Principles
Read here to understand the [Design Principles](/private-cloud-director/introduction/design-principles) that space.vars.product\_name has been built with.
# Design Principles
We've built space.vars.product\_name with these design principles.
### Familiar, Virtualization First Experience
We are a product built by VMware engineers, for virtualization administrators. We believe that virtualization must be the bedrock of any modern private cloud platform. That virtualization must be implemented as a first class citizen, and presented without complex abstractions that create complexity. We've utilized our decades of experience working at VMware to build a Private Cloud Platform with a familiar user experience compared to what you are accustomed to with your traditional virtualization interface.
### Integration and Interoperability with Existing Hardware
We aim to be broadly compatible with all major server, storage, and network investments in use by enterprises.
We believe that one of the core value propositions of private clouds is that for most businesses, there are existing hardware investments of diverse makes and generations. Often, this hardware is partially depreciated, reducing its operational costs.
We strive to allow our customers to *sweat* their existing hardware assets, and to do so, we prioritize interoperability and a wide HCL. We strive for our HCL to include all Enterprise Linux compatible hardware, and will degrade features gracefully when niche hardware capabilities are unavailable.
### Opinionated Design and UX
We are inspired by beautifully designed products that simplify end-user operations. While built on powerful open-source technologies, we aim to make these technologies simpler to learn, deploy, and operate at scale.
space.vars.product\_name will strive to simplify cloud operations and management for IT and platform engineering teams, while ensuring critical control elements are available.
For instance, to provide the best possible experience, we may abstract away lower-level APIs and controls in the underlying technology, offering a higher-level abstraction. At other times, we may make simplifying assumptions that reduce semantic complexity.
This opinionated design will be done in a manner that retains upstream API compatibility, but we may introduce additional APIs - via composition - to support our design philosophy.
We design for IT operations teams and platform engineering teams as the Administrator user, who will then optionally invite other administrator users, as well as DevOps teams and developer teams as self-service users.
### Curated Technologies
space.vars.product\_name is built using best of breed open source components under the hood, that are then coupled with our IP for enterprise readiness. Open-source projects vary widely in their maturity and readiness for enterprise usage. We select projects and technologies with a view to ensuring the following, in priority order:
1. Projects that can help us deliver a deterministic, highly available experience that enterprises can depend upon (when used with our proactive operations model)
2. Projects that have broader customer interest, vendor participation, and that are not single-vendor technologies
3. Projects that have either achieved maturity or are likely to achieve maturity due to a growing contributor base
### Cloud Completeness via Extensions
A long-standing challenge with private clouds has been their lack of the breadth of features available from public cloud providers. We are clear-eyed about the amount of effort required to match the features and services of hyperscale public clouds; yet, we are ambitious about making private clouds competitive and more attractive to developers.
Our approach to achieving Cloud Completeness will be to build foundational platform features, such as critical Infrastructure-as-a-Service (IaaS) capabilities, and to leverage the ecosystem of open-source technologies, including Kubernetes operators, to enable a broader range of higher-level application services.
# Transition from VMware
If you’re considering transitioning from VMware to an alternative virtualization stack, the space.vars.product\_name is an excellent choice. Platform9's space.vars.product\_name offers features similar to those you’ve relied on in VMware for years.
To help you get started, here’s a simple glossary mapping familiar VMware features to their counterparts in the space.vars.product\_name environment.
| **Your favorite VMware feature** | **Equivalent in PCD** |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| VMware vSphere Clusters | In space.vars.product\_acronym, [Virtualized Clusters](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint) and host aggregates serve a similar purpose to vSphere clusters. |
| VMware vCenter Server | The space.vars.product\_acronym management plane manages infrastructure resources, similar to how vCenter orchestrates VMware environments. |
| VMware ESXi Host | space.vars.product\_acronym uses KVM (Kernel-based Virtual Machine) as the underlying hypervisor. |
| VMware Distributed Resource Scheduler (DRS) | space.vars.product\_acronym Dynamic Resource Rebalancing (DRR) manages the distribution of workloads & resource consumption across hosts. |
| VMware CPU and Memory over-commitment ratios | space.vars.product\_name supports over-commitment ratio specification for CPU and Memory at a host or a host aggregate level. |
| VMware High Availability (HA) | space.vars.product\_acronym VM HA (2/4 nodes cluster) feature supports continuous availability of services. |
| VMware vMotion | space.vars.product\_acronym supports Live Migration to move instances across hosts without downtime. |
| VMware Storage vMotion | space.vars.product\_name [Storage Live Migration / Volume Retyping](/private-cloud-director/storage/volume#storage-live-migration--volume-retyping) |
| VMware Virtual or Distributed Switch (vSwitch / DVSwitch) | space.vars.product\_acronym uses Open vSwitch to provide Distributed Virtual Switch equivalent capabilities. |
| VMware Template | In space.vars.product\_acronym, templates are referred to as [Images](/private-cloud-director/images-and-image-library/image-library---images), which are used to create new virtual machines. |
| VMware Snapshot | space.vars.product\_acronym support VM Snapshots for capturing the current state of a VM. |
| VMware NSX (Networking and Security) | space.vars.product\_acronym [Networking Service](/private-cloud-director/virtualized-networking/networking-overview) offers full Software Defined Networking capabilities, including managing physical or virtual networks, routers, IP management, micro-segmentation, security policies, LBaaS, DNSaaS and more. |
| VMware Datacenter | space.vars.product\_acronym Regions, Domains, Tenants, and much more |
| VMware VMFS Datastore | space.vars.product\_acronym direct mount storage w/ storage LUNs (no shared file system) or with NFS backend |
| VMware vVol | space.vars.product\_acronym Volumes, Volume Types, Volume Snapshots |
| VMDK (Virtual Machine Disk) Format | space.vars.product\_acronym uses the QCOW2 file format for virtual machine disk images. |
| vCenter Storage Policies | space.vars.product\_acronym Storage Types and Storage Groups |
| vApps and vRealize Orchestrator (vRO) | space.vars.product\_acronym uses Terraform to automate composite cloud application deployment. |
| VMware Site Recovery Manager (SRM) | space.vars.product\_acronym supports disaster recovery through integration with most popular third-party backup and disaster recovery solutions. See [Veeam Integration with PCD](/private-cloud-director/integrations/veeam-integration-with-pcd), [Rubrik Integration with PCD](/private-cloud-director/integrations/rubrik-integration-with-pcd) Cohesity, [Commvault Integration with PCD](/private-cloud-director/integrations/commvault-integration-with-pcd). |
| VMware Resource Allocation (CPU/Memory) | space.vars.product\_acronym uses flavors to define VM configuration, including CPU and memory allocations. |
| VMware Content Library | space.vars.product\_acronym uses Image Library Service for VM templates and snapshots. |
# Pre-requisites
This document outlines the infrastructure prerequisites for setting up your private cloud. If you are looking to deploy the Self-Hosted version, please follow the [Pre-requisites](/private-cloud-director/getting-started/self-hosted/self-hosted-pre-requisites) first.
## Hypervisor Host Prerequisites
### General Hypervisor Configuration Pre-requisites
Each physical server or host that you will use as a hypervisor with space.vars.product\_name must meet the following requirements:
* x86 server - space.vars.product\_name only supports x86 server hardware today.
* Running Ubuntu 22.04 LTS (Jammy Jellyfish) server version or Ubuntu 24.04 LTS (Noble Numbat) server version for operating system.
* Download Ubuntu 22.04 -
* Download Ubuntu 24.04 -
* Each server must have hardware virtualization enabled
* an Intel processor with the Intel VT-x and Intel 64 virtualization extensions enabled or
* an AMD processor with the AMD-V and the AMD64 virtualization extensions enabled.
* Read [Hardware Virtualization Extension](/private-cloud-director/getting-started/pre-requisites/hardware-virtualization-extension) for steps to check if your server hardware has virtualization extensions enabled.
* The server must meet the [CPU Model Pre-requisites for Hypervisor Hosts](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model#cpu-model-pre-requisites-for-hypervisor-hosts).
* Your linux installation may come pre-installed with LVM package and your hypervisor operating system maybe be installed using an LVM based partition. If this is the case, you must:
* Follow [Hypervisor LVM Configuration](/private-cloud-director/getting-started/pre-requisites/hypervisor-lvm-configuration) guide to configure the required LVM filters on each hypervisor host.
* Not doing so may result in undesirable side effects listed under [Hypervisor LVM Configuration](/private-cloud-director/getting-started/pre-requisites/hypervisor-lvm-configuration#symptoms-when-not-configured).
* Each server should have the following minimum amount of resources:
1. 8 vCPUs
2. 16GB RAM
3. 250 GB storage (to include sufficient disk space for the operating system, Platform9 installer packages, logs, temporary files, virtual machine config files, virtual machine disk files when using ephemeral disks).
{% hint style="info" %}
**Note**
If you plan to only use non-ephemeral (block) storage for your virtual machines, having local storage of 100 GB at per hypervisor server level should be sufficient.
{% endhint %}
* `sudo` access enabled for the Administrator to log into the server and install the Platform9 agent
* Server `hostname` should contain at least one non-numeric character
* Make sure that the content under `/opt/pf9` is not shared across hosts. Either make this a local directory, or if using shared storage, ensure that this path mounts to a unique shared storage file share or volume that is not shared across any other hosts in your setup.
* When using the SaaS-hosted deployment model, outbound connectivity (port 443) must be enabled on each server so that the Platform9 agent can connect to the space.vars.product\_name SaaS management plane.
* In the case of a multi-domain environment, host onboarding should be done by the Administrator user in the `default` domain and not the secondary domains.
* Each server must run an NTP service (`chronyd`, `ntpd`, or `systemd-timesyncd`) **and** have its system clock actively synchronized. `pcdctl prep-node` mandatorily checks both — a host whose clock is not synchronized cannot be onboarded. Verify with `timedatectl status` and look for `System clock synchronized: yes`.
* If planning to use [VM Live Migration](/private-cloud-director/virtualized-clusters/virtualmachine/vm-migration#live-migration) feature, follow the [Live Migration Prerequisites](/private-cloud-director/virtualized-clusters/virtualmachine/vm-migration#live-migration-prerequisites)
* If planning to use the [Virtual Machine High Availability Vm Ha ](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha)feature, follow the [VM HA Prerequisites](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha#pre-requisites).
* If you plan to use the Dynamic Resource Rebalancing (DRR) feature, follow the [DRR Pre-requisites](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr#drr-pre-requisites)
### Operating System Version Compatibility
space.vars.product\_name supports both Ubuntu 22.04 LTS and Ubuntu 24.04 LTS. However, when running a mixed environment with both versions, be aware of the following limitations:
**VM Migration Between Ubuntu Versions**
Live migration of virtual machines between Ubuntu 22.04 and Ubuntu 24.04 hosts has known limitations:
* Initial VM migration from Ubuntu 22.04 to Ubuntu 24.04 typically succeeds.
* Subsequent migrations back to Ubuntu 22.04 may fail.
* Multiple round-trip migrations between versions are not supported.
**Rolling Cluster Upgrades**
When performing a rolling upgrade from Ubuntu 22.04 to Ubuntu 24.04:
* Disable VM High Availability (VM HA) before starting the upgrade.
* Disable Dynamic Resource Rebalancing (DRR) before starting the upgrade.
* Drain each host before upgrading it to Ubuntu 24.04.
* Complete all host upgrades before re-enabling VM HA and DRR This ensures workload stability during the upgrade process and prevents migration failures.
### Hypervisor Local Storage Pre-requisites
If you plan to use [Ephemeral Local Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-local-storage) for VM root disk for critical VMs in your environment, **we recommend using NFS shared storage** for your [Cluster Blueprint](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint#virtual-machine-storage-path) location. Having NFS shared storage for your VM storage path will make it easy to ensure that you have sufficient storage space for your VM root disks, and that you can expand this space quickly in the future, if required.
Regardless of whether you use NFS shared storage or local storage, you must ensure that there's sufficient local disk space at per hypervisor host level to store the virtual machine root disk files for the maximum number of VMs that will run on a hypervisor host at any given point.
The recommended minimum storage per hypervisor host:
* **250GB** of minimum disk size
* This storage will be used by operating systems, space.vars.product\_name service components, log files and **virtual machine root disk files** for VMs using ephemeral storage.
If you plan to use [Ephemeral Shared Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-shared-storage) or [Block Storage Volumes](/private-cloud-director/storage/volume) for VM root disk, then per hypervisor host local disk requirements will be lower:
* **100GB** of local disk space
* This storage will be used by operating system, space.vars.product\_name service components, and log files.
### Swap Configuration
We recommend that you configure swap memory on all hosts that you add to your space.vars.product\_name setup, to enhance system performance and to effectively manage memory-intensive VM workloads. Follow the steps in [Configure Swap](/private-cloud-director/getting-started/pre-requisites/configure-swap-ubuntu) to configure your swap memory and the swappiness value for your host.
### Networking Prerequisites
Read [Overview & Architecture](/private-cloud-director/virtualized-networking/networking-overview) for a detailed understanding of networking in space.vars.product\_name.
All hypervisor hosts should have a minimum of one network interface, and ideally four network interfaces to enable redundancy across network interface failure.
Use of bonded network interfaces is recommended to ensure high availability in case of a physical network interface failure.
A typical configuration would look like:
* bond0 mapped to two network adapters: eth0 and eth1
* bond1 mapped to two network adapters: eth2 and eth3
space.vars.product\_name allows you to designate separate network interfaces for the following types of traffic:
* Management network
* Image library I/O network
* VM console network
* Virtual network tunnels
#### Outbound Connectivity Requirements
You would need to configure outbound access on port 443 from your hosts for the below domain names to ensure they can be onboarded to the space.vars.product\_name management plane.
1. space.vars.product\_name management plane url is accessed over port 443.
2. For `pcdctl` CLI download on hosts,
3. APT sources list for installing packages on the Ubuntu host using `pcdctl prep-node` :
1.
2.
3.
4.
5.
#### Connectivity via http(s) proxy server
When outbound connectivity needs to be routed via a proxy server, ensure that the `/etc/environment` file has the http\_proxy, https\_proxy and no\_proxy variables defined as follows:
{% tabs %}
{% tab title="Bash" %}
```bash
https_proxy=http://pf9:squid@squid.pf9.io:3128
http_proxy=http://pf9:squid@squid.pf9.io:3128
no_proxy=127.0.0.1,localhost,..
```
{% endtab %}
{% endtabs %}
The format of a typical proxy server URL is `[][:@]:` .
Also, the `no_proxy` variable should include IP addresses for which the traffic should not be routed via the proxy, e.g. other servers that would be set up as Hypervisor, Image Library or Storage roles.
Additionally, ensure that apt uses the proxy server to fetch packages. This can be configured by updating the `/etc/apt/apt.conf.d/proxy.conf` file with the following entries
{% tabs %}
{% tab title="Bash" %}
```bash
Acquire::http::Proxy "http://pf9:squid@squid.pf9.io:3128";
Acquire::https::Proxy "http://pf9:squid@squid.pf9.io:3128";
```
{% endtab %}
{% endtabs %}
### VNC Console Prerequisites
The VNC console service is added to all hypervisor hosts as part of configuration of hypervisor role. This allows you to access the VMs on the host from your web browser. The following prerequisites must be met for this feature to work
* Ensure that the port `6080` is open on each host.
* Ensure that you are on the same network as the host.
* To access the console using a public or floating IP address, or a domain name, configure the corresponding option in the cluster blueprint. Ensure that proper routing and DNS resolution are configured for the specified IP or domain. This setting applies to all hypervisor hosts in the region.
{% hint style="info" %}
**Console security model**
The underlying QEMU VNC sockets on each hypervisor are bound to the hypervisor's IP address and are reachable from the network if routing exists. VNC sessions are encrypted with TLS and authenticated with an automatically generated per-session password — no user configuration required. All authenticated browser console access flows through the noVNC service on port `6080`. TLS material for the libvirt VNC server is managed automatically under `/etc/pki/libvirt-vnc/`.
{% endhint %}
## Persistent / Block Storage Prerequisites
space.vars.product\_name supports a wide variety of enterprise storage solutions. Verify you have access to the administrative console of your storage solution and can look up the required configuration information from your admin console.
* Read more in the [Storage Overview](/private-cloud-director/storage/storage-overview) article about types of storage supported by space.vars.product\_name.
* For [block storage](/private-cloud-director/storage/volume), see the list of [supported block storage drivers](/private-cloud-director/storage/block-storage/supported-storage-drivers)
* space.vars.product\_name expects each hypervisor that connects to iSCSI storage must present **one unique iSCSI Qualified Name (IQN)**. Duplicate IQN can exist across hypervisor hosts when multiple hosts boot with an identical IQN, often because their OS image was cloned. Please refer to the [knowledge base article](https://platform9.com/docs/private-cloud-director/kb/how-to-rename-iscsi-initiator-names-to-the-existing-hypervisors-) to address duplicate IQNs.
### Temporary Volume Storage Requirements
Anytime you create a volume using an image, the directory `/opt/pf9/pf9-cindervolume-base/state/conversion` is used as a temporary staging area to store the image file while the conversion is happening. Therefore, for any hosts with persistent storage role assigned, this directory must have sufficient storage space to **support the cumulative size based on the maximum 'uncompressed' image size in your environment and the maximum number of concurrent new volume creations using images** you expect to happen in your setup. You can estimate the uncompressed size of a qcow2 image by doubling it's current disk size.
Also see here [latest compatibility matrix of Cinder storage drivers and devices](https://docs.openstack.org/cinder/latest/reference/support-matrix.html#driver-support-matrix).
## Image Library Prerequisites
The Image Library service manages virtual machine images in the space.vars.product\_name environment.
To enable its proper operation, the following prerequisites must be met on **all hosts with Image Library role assigned**:
* The Image Library service runs on port `9494`. **Ensure this port is open** to allow image operations, specifically image uploads, edit or delete operations from web browsers and CLI clients.
* Using shared storage for the image library service is required to create a highly available image library setup.
* Image uploads and deletes using the CLI require the use of the [Image Library Admin Endpoint](/private-cloud-director/images-and-image-library/image-library---images#image-library-admin-endpoint), and the client machine being used to upload the image must have network reachability to the IP address of the image library host that is serving as the admin endpoint.
### Image Library Temporary Storage Requirements
* Anytime a new image is being uploaded to your image library, the directory **`/var/lib/nginx`** is used as a temporary staging area to store the image file while the upload is happening. This means at any given point, **sufficient storage space** should be available under this directory to **support the cumulative size based on the maximum image size in your environment and the maximum number of concurrent image upload** operations you expect to happen to the image library.
* We therefore recommend using shared storage (eg NFS) for this directory if you plan to use images with disk size greater than approximately 50GB.
### Browser Connectivity
The host that you've assigned the image library role (the image library host) must be accessible via a web browser from the machines your users will use to access space.vars.product\_name UI . This requirement is necessary for:
* Uploading images through the space.vars.product\_name UI.
* Verifying and accepting image library self-signed certificates.
### Self-Signed Certificates
The space.vars.product\_name image library service uses self-signed certificates today to secure the communication for image uploads. Since browsers and CLI tools only trust publicly verified certificates, users must manually accept the self-signed certificate before they can upload any images to the image library service.
To accept the self-signed certificate:
* Navigate to the image library endpoint in a browser.
* Click Access & Security Menu -> API Access -> and look for glance-cluster.
* Accept the insecure certificate when prompted.
## Load Balancer As a Service (LBaaS) Prerequisites
Follow the [Load balancer as a service Prerequisites](/private-cloud-director/virtualized-networking/load-balancer#prerequisites) if you plan to use the space.vars.product\_name built-in LBaaS component for your applications.
## Kubernetes Prerequisites
Read [Kubernetes Pre-requisites](/private-cloud-director/kubernetes-clusters/k8s-pre-requisites) for setting up a Kubernetes cluster in space.vars.product\_name
# Configure Swap
This document describes the steps to configure swap memory and swapiness on a Ubuntu server.
## Configure Swap Memory
We recommend configuring your swap memory size to be **1.5x** **of your host physical RAM** for best performance.
There are two ways to configure swap memory for your host. You can either create a swap file in an existing partition, or allocate a dedicated disk partition for swap.
### Configure Swap Memory using Swapfile
Create a new Swapfile with a specific size using `fallocate`.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo fallocate -l 200G /swapfile.img
```
{% endtab %}
{% endtabs %}
Modify the Swapfile permissions to allow only the root user to read and write changes on the file.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo chmod 0600 /swapfile.img
```
{% endtab %}
{% endtabs %}
Format the file as swap using `mkswap`.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo mkswap /swapfile.img
```
{% endtab %}
{% endtabs %}
### Configure Swap Memory using Disk Partition
Convert your designated block storage partition to swap. In the example below, `/dev/vdb1` will be dedicated as your swap partition.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo mkswap /dev/vdb1
```
{% endtab %}
{% endtabs %}
### Enable Swap Memory
Once you have configured your swap memory, the next step is to enable swap using the `swapon` command.
If using swapfile:
{% tabs %}
{% tab title="Bash" %}
```bash
sudo swapon /swapfile.img
```
{% endtab %}
{% endtabs %}
If using dedicated partition:
{% tabs %}
{% tab title="Bash" %}
```bash
sudo swapon /dev/vdb1
```
{% endtab %}
{% endtabs %}
Finally, verify that the swap partition is active.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo swapon -s
```
{% endtab %}
{% endtabs %}
### [Make the Swap File Permanent](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-22-04#step-5-making-the-swap-file-permanent)
If using swap file, you will need to add the swap file to your `/etc/fstab` file, so that the swap settings are retained across server reboot.
Backup your fstab file first.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo cp /etc/fstab /etc/fstab.bak
```
{% endtab %}
{% endtabs %}
Now add the swap file to fstab.
{% tabs %}
{% tab title="Bash" %}
```bash
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
{% endtab %}
{% endtabs %}
## Configure Swappiness
### What is Swappiness
Swappiness is a linux kernel parameter that determines the system's tendency to move data from physical RAM to the swap space on a disk. It's a value between 0 and 100.
Default swappiness value for Ubuntu is 60.
### Recommended Swappiness Value
For space.vars.product\_name hypervisors, **we recommend setting the swappiness value to 10**, to minimize swapping and favor keeping processes in RAM to avoid performance degradation.
A swappiness value of 10 means swapping is only used as a last resort when RAM is nearly exhausted, prioritizing the performance and responsiveness of VMs
Setting swappiness higher increases the likelihood of swap usage, which is not ideal for hypervisors due to the significant latency swap imposes on VM workloads.
### Configure Swappiness
Use the following command to configure swappiness on your Ubuntu server.
Step 1 - Edit `/etc/sysctl.conf`
Step 2 - Add `vm.swappiness = 10`
Step 3 - Apply the changes by running `sudo sysctl -p`
Step 4 - Confirm that the changes are applied by running `cat /proc/sys/vm/swappiness`
# Hardware Virtualization Extension
This document describes the steps to validate that your server hardware that you plan to use with space.vars.product\_name has hardware and optionally I/O virtualization extensions enabled.
## Hardware Virtualization
space.vars.product\_name uses KVM hypervisor underneath which relies on hardware virtualization extensions (Intel VT-x or AMD-V) for full virtualization.
Specifically, space.vars.product\_name requires that your physical servers run with:
* an Intel processor with Intel VT-x and Intel 64 virtualization extensions enabled, or
* an AMD processor with AMD-V and AMD64 virtualization extensions enabled
Follow the steps below to check if your server hardware comes with hardware virtualization extensions and if they are enabled.
Run the following command to validate that CPU virtualization extensions are available on your server:
{% tabs %}
{% tab title="Bash" %}
```bash
$ grep -E 'svm|vmx' /proc/cpuinfo
```
{% endtab %}
{% endtabs %}
If the above command returns any output, that means your server hardware contains the hardware virtualization extensions.
In the output, look for a `vmx` entry, indicating an Intel processor with the Intel VT-x extension enabled, or an `svm` entry, indicating an AMD processor with the AMD-V extensions enabled.
In some cases, manufacturers may have disabled the virtualization extensions in the server BIOS settings. If the output of the above command is empty, or if full virtualization does not work, find your server and processor specific instructions on enabling the extensions in your server BIOS configuration.
## I/O Virtualization
Both Intel and AMD have hardware virtualization extensions available to allow virtual machines to have direct access to hardware devices. This is specifically important to enable passthrough access to GPU devices, when using GPUs with space.vars.product\_name.
If you plan to consume GPUs via GPU passthrough technology, then the following virtualization extension must be enabled:
* VT-d extensions enabled for an Intel processor, or
* AMD IOMMMU extensions enabled for an AMD processor
To check if the I/O virtualization extensions are enabled for your physical server, run the following command:
{% tabs %}
{% tab title="Bash" %}
```bash
$ ls /sys/class/iommu/
```
{% endtab %}
{% endtabs %}
Examine the output.
* If the directory contains entries like `dmar0`, `dmar1` (for Intel systems), it indicates that IOMMU (and thus VT-d) is enabled
* If the directory contains entries like `amd-iommu-0` or similar, it suggests that the AMD IOMMU is recognized and potentially enabled by the kernel.
If the directory is empty, IOMMU might be disabled or not recognized.
# Hypervisor LVM Configuration
### Overview
PCD compute nodes (hypervisors) have LVM configured and running on the hypervisor node, must follow the guidelines below to properly configure LVM filters to prevent system hangs and performance issues.
This document outlines the required LVM filter configuration.
### Check if LVM Package is Installed
Run the following command to check if LVM package is installed on your system.
```bash
dpkg -l | grep lvm
ii libllvm15:amd64 1:15.0.7-0ubuntu0.22.04.3 amd64 Modular compiler and toolchain technologies, runtime library
ii liblvm2cmd2.03:amd64 2.03.11-2.1ubuntu4 amd64 LVM2 command library
ii lvm2 2.03.11-2.1ubuntu5 amd64 Linux Logical Volume Manager
```
If LVM is installed, you will see output similar to above. In that case, follow the steps below to set LVM filters.
### Identify Physical Disk Devices On Your System
Run the command below to identify all the block devices mounted on this host. NOTE however that this command will also list any VM disks as well.
```bash
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,VENDOR,MODEL | grep disk
sda 1.7T disk HPE LOGICAL VOLUME
sdb 1.0T disk HPE LOGICAL VOLUME
sdg 49G disk 3PARdata VV
sdk 49G disk 3PARdata VV
sdl 49G disk 3PARdata VV
sdm 49G disk 3PARdata VV
```
Identify from the output above the **subset of entries that are the physical disks for your hypervisor host**.
In this example, you would only pick `sda` and `sdb`. The other four devices are volumes attached to VMs running on this host.
### Add Or Enable LVM Filters For All Disks
Edit `/etc/lvm/lvm.conf` and locate the devices section.
Then edit the `filter` and `global_filter` values and add the **entries for the physical disks for your hypervisor host** that you identified above.
Note that this is a regular expression, where 'a' stands for accepting the path and 'r' stands for rejecting the path in the syntax below.
This example
```bash
devices {
filter = [ "a|^/dev/sda|", "a|^/dev/sdb|", "r|.*|" ]
global_filter = [ "a|^/dev/sda|", "a|^/dev/sdb|", "r|.*|" ]
}
```
### Validate that the LVM Filters are Active
```bash
sudo lvm dumpconfig devices | grep -E "filter|global_filter"
```
### Symptoms When Not Configured
If LVM is installed on your hypervisor but the LVM filters are not configured properly, here are some of the side effects and symptoms that you may observe on your space.vars.product\_name hosts:
* QEMU processes stuck in uninterruptible sleep (D state)
* VM operations hang or timeout
* Slow system responsiveness on the host
* High I/O wait times during LVM operations
* VMs fail to start or become unresponsive
# Getting Started
This document provides steps to get started with your space.vars.product\_name setup.
If you haven't yet, make sure to follow the [Pre Requisites](/private-cloud-director/getting-started/pre-requisites) to get ready for install.
## Step 1 - Setup your Private Cloud Director Instance
The commercial version of space.vars.product\_name offers two deployment options: **SaaS or Self Hosted**. Or you can use the free community edition of space.vars.product\_name for your home lab / test setup.
### 1. SaaS Hosted
With SaaS Hosted model, we setup and host space.vars.product\_name in the cloud for you and you consume it as a SaaS service. **Your infrastructure still remains in your data center.** The SaaS hosted version of space.vars.product\_name communicates remotely with your infrastructure to get your private cloud setup and running. This is generally the best option for organizations that prefer the ease of use of SaaS and don't want to manage the technical complexity of hosting the product themselves.
### 2. Self Hosted
With Self Hosted model, you can host space.vars.product\_name on your on hardware in your data center or a co-location hosted data center. This is generally the best option for organizations with extra compliance and security requirements where SaaS hosting is not a viable option.
Follow [Self Hosted Install](/private-cloud-director/getting-started/self-hosted/self-hosted-install) for steps to install the self hosted management plane.
### 3. Community Edition
space.vars.product\_name also offers a community edition that is completely free to download and use.
Follow [Community Edition](/private-cloud-director/getting-started/getting-started-with-community-edition) guide for steps to install space.vars.product\_name Community Edition.
### 4. Platform9 OS (Beta)
Platform9 OS bundles Rocky Linux by CIQ (RLC) and space.vars.product\_name into a single ISO-based installation. It is designed for organizations that want a fully integrated, appliance-style deployment where the operating system and space.vars.product\_acronym are installed together in one workflow — no separate OS provisioning required.
{% hint style="warning" %}
Platform9 OS is currently in closed beta. Access is by invitation only. Contact your Platform9 account manager to request access.
{% endhint %}
Follow [Platform9 OS Beta Installation](/private-cloud-director/getting-started/platform9-os-beta-install) for steps to install using the ISO-based installer.
## Step 2 - Create Cluster Blueprint & Virtualized Cluster
Once you have a space.vars.product\_name instance setup, next step is to start onboarding your hypervisors. Follow the steps below to create a [Cluster Blueprint](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint) and start onboarding hypervisors.
1. Create a virtualized cluster blueprint. Follow [Virtualized Cluster Blueprint](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint) article for more information on what cluster blueprints are and how to create one.
2. Once the cluster blueprint is created, you are now ready to create one or more virtualized clusters.
3. Follow [Create a Virtualized Cluster](/private-cloud-director/virtualized-clusters/virtualized-cluster#create-a-virtualized-cluster) for steps to create a cluster.
4. Your first virtualized cluster is now created.
## Step 3 - Add Hosts to Cluster
1. The next step is to start onboarding your physical servers or hosts and adding them to the cluster.
1. For SaaS hosted management plane - follow [Add a Host (SaaS Deployment)](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster#add-a-host-saas-deployment)
2. For Self Hosted or Community Edition - follow [Add a Host (Self-Hosted & Community Edition Deployment)](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster#community-edition-deployment)
2. Now you will have a number of hosts showing up as unauthorized in your space.vars.product\_name UI.
3. Follow [Authorize Host And Assign Roles](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster#authorize-host-and-assign-roles) to authorize one or more roles for the hosts and assign them to the cluster.
## Step 4 - Create Images and Networks
1. Follow [Image Library Images](/private-cloud-director/images-and-image-library/image-library---images) to create your first image in your image library.
2. Create at least one [Virtual Network](/private-cloud-director/virtualized-networking/networks-and-ports) or [Physical Network](/private-cloud-director/virtualized-networking/physical-network).
## Step 5 - Create Virtual Machines
1. Read [Virtual Machines](/private-cloud-director/virtualized-clusters/virtualmachine) for steps to provision your first virtual machine on your cluster.
## Step 6 - Create Kubernetes Cluster
1. Read [Getting Started with Kubernetes](/private-cloud-director/kubernetes-clusters/getting-started-with-kubernetes-in-pcd) to create your first Kubernetes cluster using space.vars.product\_name.
# Community Edition
Understand what Community Edition is, how it’s different, and where to start.
{% hint style="danger" %}
**Community Edition is not for production workloads.** Use it for labs, evaluation, and learning only.
{% endhint %}
{% hint style="info" %}
**New release!**
For details on the latest release of space.vars.product\_name Community Edition, see the [Release Notes](/release-notes/april-2026-release).
{% endhint %}
## Community Edition At a Glance
Community Edition (CE) is the community-supported, free, full-featured version of space.vars.product\_name.
You get the same core virtualization experience as the commercial product.
CE is different in how it’s deployed and supported:
* **Single-host management plane.** CE runs the management plane on one machine. Note that this is only the limitation on how the management plane is provisioned. There is no limit on the number of hypervisors you can add to the CE management plane.
* **You bring the hypervisors.** Workloads always run on separate hypervisor hosts.
* **Community support.** Use and these docs.
{% hint style="info" %}
If you want a Platform9-managed control plane, use SaaS Hosted. If you need a customer-managed highly available control plane, use [Self Hosted](/private-cloud-director/getting-started/self-hosted).
See [Getting Started](/private-cloud-director/getting-started/getting-started) for a quick comparison.
{% endhint %}
***
## How Community Edition Works
CE is two system types working together:
* **CE host (control plane)**
* **Hypervisor hosts (run VMs)**
### Community Edition Host (Control Plane)
The **CE host** is where you manage the platform.
It:
* Serves the web UI
* Exposes REST APIs
* Monitors hosts and VMs
* Orchestrates lifecycle operations (create, resize, migrate, HA, and more)
* Hosts Grafana dashboards
It does **not** run your VMs.
### Hypervisor Hosts (Compute)
**Hypervisor hosts** run the virtual machines.
Each hypervisor runs a **Platform9 host agent**. The agent:
* Connects the host to the CE control plane
* Executes lifecycle actions
* Sends health and metrics back to CE
### Shared Storage (Optional, But Recommended)
Shared storage (NFS, iSCSI, and other supported backends) enables:
* **Persistent volumes** for VM disks
* **Live migration** between hypervisors
* **VM High Availability (VM HA)** that can restart VMs on another host
If you use **only local storage** on hypervisors:
* Live migration requires VM shutdown.
* VM disks stay tied to the original host.
* VM HA can’t protect against host loss for those VMs.
***
## Get Started
1. Validate requirements in [Prerequisites](/private-cloud-director/getting-started/getting-started-with-community-edition/prerequisites).
2. Run the standard install in [Install](/private-cloud-director/getting-started/getting-started-with-community-edition/install).
3. If you need custom CIDRs, domains, or other changes, read [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation) **before** installing.
4. If anything fails, use [Common Issues](/private-cloud-director/getting-started/getting-started-with-community-edition/common-issues).
{% hint style="info" %}
Want a walkthrough instead of reference docs? Use the tutorial: [Beginner's Guide to Deploying PCD Community Edition](/private-cloud-director/tutorials/beginners-guide-to-deploying-pcd-community-edition).
{% endhint %}
## Operations
* Backup strategy and procedures: [Backup and Restore CE](/private-cloud-director/getting-started/getting-started-with-community-edition/backup-and-restore-ce)
# Prerequisites
Host, network, and storage requirements for installing Community Edition.
{% hint style="danger" %}
**Community Edition is not for production workloads.** Use it for labs, evaluation, and learning only.
{% endhint %}
## Quick Start Checklist
Use this if you want the minimum viable lab setup.
* [ ] Community Edition (CE) Host: [Ubuntu 24.04](https://cloud-images.ubuntu.com/releases/noble/release/) VM with 8 CPUs, 32GB RAM, 100+ GB free space
* [ ] Hypervisor host: [Ubuntu 24.04](https://cloud-images.ubuntu.com/releases/noble/release/) VM with 8+ CPUs, 16+ GB RAM, 100+ GB free space
* [ ] Both the CE host and hypervisor host(s) can reach the internet
* [ ] You can access both CE and the hypervisor(s) via SSH (port 22)
* [ ] You can access the CE host via HTTPS (port 443)
* [ ] Install Community Edition on the CE host (see [Install](/private-cloud-director/getting-started/getting-started-with-community-edition/install))
```bash
curl -sfL https://go.pcd.run | bash
```
## Detailed Prerequisites
Community Edition (CE) uses two system types:
* **CE host**: runs the control plane and web UI.
* **Hypervisor hosts**: run your VMs.
### Community Edition Host (CE Host)
This host runs the space.vars.product\_name control plane and web interface.
#### Operating system
* **Supported:** [Ubuntu 24.04 AMD64 cloud image](https://cloud-images.ubuntu.com/releases/noble/release/)
* **Not supported:** Minimal cloud image (missing required packages, such as `crontab`)
* **Other distributions:** Not supported by the installer
#### Sizing
* **CPU:** 8 logical CPUs, 12+ for Kubernetes workloads
* **RAM:** 28GB minimum, 32GB recommended
* **Disk:** 100GB minimum free space, SSD recommended
* **CPU architecture:** Intel Nehalem / AMD Bulldozer or newer (x86-64-v2)
#### Kubernetes management plane (optional)
If you enable the Kubernetes management plane during installation, the CE host requires:
| Resource | Minimum | Recommended |
| -------- | --------------- | --------------- |
| **CPU** | 12 logical CPUs | 16 logical CPUs |
| **RAM** | 32 GB | 48 GB |
The installer verifies the minimum requirements before proceeding. Disable the Kubernetes management plane to use the base CE host requirements.
#### Network
* **Internet access**
* **IPv6:** Must be enabled, but doesn't need an address
* **Firewalld:** Must be stopped and disabled (if installed)
```bash
# Stop the firewalld service
sudo systemctl stop firewalld
# Prevent it from starting on boot
sudo systemctl disable firewalld
```
Ports and outbound access
**Ports open locally (CE host):**
* 443, 2379, 2380, 3306, 4194, 5395, 5672, 5673, 6264, 8023, 8158, 8285, 8558, 9080, 10250, 10255
**Outbound access:**
* TCP 53, 443
* UDP 53, 123
### Hypervisor Hosts
These hosts run your actual VMs.
#### Operating system
* **Supported:** [Ubuntu 24.04 AMD64](https://cloud-images.ubuntu.com/releases/noble/release/) cloud image
* **Not supported:** Minimal cloud image (missing required packages, such as `crontab`)
* **Other distributions:** Not supported by the installer
#### Sizing
* **CPU:** 8 logical minimum, 16+ recommended
* **RAM:** 16GB minimum, 32GB+ recommended
* **Disk:** 100GB minimum free space, 200GB+ recommended (shared storage is better)
#### Nested virtualization (if hypervisors are VMs)
If your hypervisor hosts run as VMs, nested virtualization must be enabled.
**VMware vSphere**
* For hypervisor hosts running as VMs on vSphere:
* Enable hardware virtualization on the vCPUs before installing Ubuntu
* Allocate at least 115GB thin-provisioned disk (to allow for 100+ GB free space after OS)
* Accept promiscuous mode, MAC address changes, and forged transmits on the vSwitch (not at the port group!) to allow nested VMs running on the hypervisor VM to access the external network
**Proxmox**
* Enable CPU type "host" or "kvm64"
* Enable nested virtualization
{% hint style="info" %}
If hypervisor hosts are running as VMs, nested virtualization must be enabled and visible to the hypervisor host operating system.
{% endhint %}
#### Nested virtualization check
```bash
# Run this on the hypervisor host
egrep "svm|vmx" /proc/cpuinfo
# Should show output with vmx (Intel) or svm (AMD) flags
# No output = nested virtualization not enabled
```
### Storage Backend
You need shared storage for persistent VM disks and live migration. Ephemeral disks work without shared storage and are stored locally on each hypervisor host.
Options:
1. **NFS (Easiest for Testing)**
* Minimum: 100GB+ available on NFS mount point
* Best for: Lab environments, proof-of-concept
* Performance: Good for testing, adequate for light workloads
2. **External Storage Array**
* Minimum: Depends on your needs
* Best for: dev/test environments, extended proof-of-concept
* Performance: Excellent
3. **Local Storage**
* Uses hypervisor local disk
* Best for: Ephemeral VMs, temporary workloads
* Limitation: VM live migration and VM HA are not supported
See [Storage Overview](/private-cloud-director/storage/storage-overview) for detailed configuration.
### Network Requirements
#### Default Kubernetes CIDRs
* Services: `10.21.0.0/16`
* Pods: `10.20.0.0/16`
{% hint style="info" %}
**Check for conflicts:** If your network uses these ranges, see [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation) to change them BEFORE installing.
{% endhint %}
#### Required external URLs
The CE installer must reach these endpoints. If you’re behind a firewall, whitelist them.
* `go.pcd.run` - Installer script
* `quay.io`, `registry.k8s.io` - Container images
* `github.com` - Software downloads
* `gcr.io` - Required if the Kubernetes management plane is enabled
* [Complete list](/private-cloud-director/getting-started/getting-started-with-community-edition/common-issues#required-ports-and-external-urls) of required ports and URLs
{% hint style="warning" %}
**Proxy environments:** The Kubernetes management plane can be installed in proxy environments, but provisioning Kubernetes clusters through it is not supported.
{% endhint %}
# Install
Default installation steps for Community Edition, plus first-login and basic operations.
{% hint style="danger" %}
**Community Edition is not for production workloads.** Use it for labs, evaluation, and learning only.
{% endhint %}
{% hint style="info" %}
For a guided walkthrough, use [Beginner’s Guide to Deploying PCD Community Edition](/private-cloud-director/tutorials/beginners-guide-to-deploying-pcd-community-edition).
{% endhint %}
## Install Community Edition
This installs the Community Edition (CE) control plane on a single **CE host**. You then onboard one or more **hypervisor hosts** to run VMs.
{% hint style="info" %}
Read [Prerequisites](/private-cloud-director/getting-started/getting-started-with-community-edition/prerequisites) first. See [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation) if you need custom CIDRs or domains.
{% endhint %}
### Outcome
After install, you can:
* Log in to the web UI
* Onboard hypervisor hosts
* Create networks, images, and VMs
* View metrics in the web UI or Grafana
### Timeline
Typical runtime is \~45 minutes. Slow systems or slow registries can push it to \~90 minutes.
* 5–10 min: system prep and prerequisite checks
* 10–15 min: K3s cluster creation
* 20–30 min: space.vars.product\_acronym services deployment to the K3s cluster
* \~5 min: verification and credential output
### Deploy the CE host
Run the installer on the CE host as a user with `sudo` access.
```bash
curl -sfL https://go.pcd.run | bash
```
{% hint style="info" %}
By default in Ubuntu, all users in the `sudo` group have the ability to sudo. Group membership can be validated with the command `groups ` and sudo permissions can be verified with `sudo -l -U ` .
{% endhint %}
The installer will:
* Ask you to accept the [Community Edition EULA](https://platform9.com/ce-eula)
* Ask whether to enable the Kubernetes management plane (optional - see [Kubernetes management plane prompt](#kubernetes-management-plane-prompt))
* Run prerequisite checks
* Deploy the CE management plane
* Print the user interface web address and provide admin credentials
If prerequisite checks fail, see [Common Issues](/private-cloud-director/getting-started/getting-started-with-community-edition/common-issues). If installation fails, the installer can optionally upload a support bundle.
#### Kubernetes management plane prompt
The installer asks whether to enable the Kubernetes management plane, which lets you provision managed Kubernetes clusters. This is opt-in — the default is No.
Enabling it:
* Adds approximately 10 minutes to install time
* Requires a larger CE host - 12 CPUs and 32 GB RAM minimum (see [Prerequisites](/private-cloud-director/getting-started/getting-started-with-community-edition/prerequisites#kubernetes-management-plane-optional))
To skip the prompt for automation, pre-export the variable before running the installer:
```bash
# Enable Kubernetes management plane
export ENABLE_K8S=true
# Disable Kubernetes management plane
export ENABLE_K8S=false
```
Installation example
```bash
ubuntu@ce-host:~$ curl -sfL https://go.pcd.run | bash
Private Cloud Director Community Edition Deployment Started...
By continuing with the installation, you agree to the terms and conditions of the
Private Cloud Director Community Edition EULA.
Please review the EULA at: https://platform9.com/ce-eula
Do you accept the terms of the EULA? [Y/N]: y
Private Cloud Director can install a Kubernetes management plane
that lets you provision managed Kubernetes clusters.
This is optional and adds ~10 minutes to install time and requires additional
CPU / memory (see [Prerequisites](prerequisites.md)).
Enable Kubernetes management plane? [y/N]: y
Finding latest version... Done
Downloading artifacts... Done
Configuring system settings... Done
Installing artifacts and dependencies... Done
Configuring Docker Mirrors... Done
SUCCESS Configuration completed
INFO Verifying system requirements...
✓ Architecture
✓ Disk Space
✓ Memory
✓ CPU Count
✓ OS Version
✓ Swap Disabled
✓ IPv6 Support
✓ Kernel and VM Panic Settings
✓ Port Connectivity
✓ Firewalld Service
✓ Default Route Weights
✓ Basic system services
Completed Pre-Requisite Checks on local node
SUCCESS Cluster created successfully
INFO Starting PCD management plane
SUCCESS Certificates generated
SUCCESS Base infrastructure setup complete
SUCCESS Installed Kubernetes management plane
SUCCESS pcd-virt deployment now complete
SUCCESS Final touches...
Private Cloud Director (Community Edition) deployment complete!
How would you rate your Private Cloud Director (Community Edition) installation experience?
(1-5, press Enter to skip): 5
------------- deployment details ---------------
fqdn: pcd.pf9.io
region: pcd
deployment status: ready
region health: ✅ Ready
version: PCD 2026.4-116
-------- region service status ----------
desired services: 56
ready services: 56
Login Details:
URL: https://pcd.pf9.io
email: admin@airctl.localnet
password: wVXJbocBrZaqcjtS
See documentation at https://pcd.run for next steps.
Join the support community at https://reddit.com/r/platform9
ubuntu@ce-host:~$
```
#### Enable Kubernetes management plane on an existing CE installation
To enable the Kubernetes management plane on an existing CE deployment:
{% hint style="info" %}
The base CE installation must be fully deployed and healthy before running this command.
{% endhint %}
```bash
sudo /opt/pf9/airctl/airctl start --config /opt/pf9/airctl/conf/airctl-config.yaml --k8s-only
```
This takes approximately 10 minutes.
To verify the installation succeeded:
```bash
kubectl get pods -n kaapi
```
All pods should show `Running`.
{% hint style="warning" %}
The CE host must meet the minimum requirements for the Kubernetes management plane: 12 CPUs and 32 GB RAM. See [Prerequisites](/private-cloud-director/getting-started/getting-started-with-community-edition/prerequisites#kubernetes-management-plane-optional).
{% endhint %}
### Access the web UI
For default installations, the web user interface can be accessed at `https://pcd.pf9.io`. That name must resolve from your browser.
{% hint style="info" %}
If you've [customized the Community Edition FQDN](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation#deployment-fqdn-fully-qualified-domain-name), then replace `pcd.pf9.io` with that value.
{% endhint %}
#### Option 1: Local hosts file (single machine)
Use this for quick testing, as only your computer will be able to resolve the URL to the user interface.
**Linux/MacOS**
```bash
# Replace with your CE host IP
echo " pcd.pf9.io" | sudo tee -a /etc/hosts
# Example
echo "192.168.1.100 pcd.pf9.io" | sudo tee -a /etc/hosts
```
**Windows**
1. Open Notepad as Administrator. From the Windows Start menu, right-click the Notepad icon and choose "Run as Administrator".
2. Open `C:\Windows\System32\drivers\etc\hosts`.
3. Add the following, but be sure to replace the example IP. Save and close the file afterwards.
```
192.168.1.100 pcd.pf9.io
```
#### Option 2: Real DNS (team access)
Create A records in your DNS server:
```
pcd.pf9.io ->
```
### Log in
Open the user interface URL in a browser (the default is [pcd.pf9.io](https://pcd.pf9.io/)). Accept the self-signed certificate on first visit.
* Select **Use local credentials**.
* Use the `admin@airctl.localnet` credentials printed by the installer.
* Keep **Domain** as `default`.
* Do not check "I have an MFA token".
{% hint style="info" %}
SSO & MFA are unavailable until configured. Errors with an SSO or MFA login attempt will fail until they are configured.
{% endhint %}
Private Cloud Director Community Edition local credentials login screen
### Admin credentials and Grafana access
Do not rename the default admin user, `admin@airctl.localnet`. Do not change its password.
This account is used internally by space.vars.product\_name services.
Create separate users for administrator or self-service access.
{% hint style="warning" %}
The default admin account is also used for Grafana authentication. Changing the default admin email or password can break Grafana access.
{% endhint %}
### Next steps
1. See the Getting Started guide at [Getting Started](/private-cloud-director/getting-started/getting-started#step-2-create-cluster-blueprint-and-virtualized-cluster)
2. Follow the rest of the Getting Started guide to onboard hypervisor hosts, create networks, upload images, and create virtual machines.
***
## Operations and troubleshooting
Validate the installation
Your deployment is healthy if the region is marked as `Ready`. Ready services should match desired services.
```bash
/opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
Example output:
```bash
root@pcd-ce:~# /opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
------------- deployment details ---------------
fqdn: pcd.pf9.io
region: pcd
deployment status: ready
region health: ✅ Ready
version: PCD 2026.4-108
-------- region service status ----------
desired services: 54
ready services: 54
```
Retrieve admin credentials
```bash
/opt/pf9/airctl/airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml
```
Logs and service names
**Hypervisor host logs**
* `/var/log`
* `/var/log/pf9`
**CE host logs**
* `/var/log/pf9/fluentbit/ddu.log` (JSON; includes Kubernetes pod logs)
You can also use `kubectl logs` to fetch pod logs.
**Hypervisor host services**
Service names start with `pf9`. Examples:
* `pf9-hostagent`
* `pf9-imagelibrary`
* `pf9-ostackhost`
Grafana credentials
Grafana uses the same admin credentials printed at install time.
* Username: `admin@airctl.localnet`
* Password: retrieve from the CE host:
```bash
/opt/pf9/airctl/airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml
```
Default Grafana URL:
* `https://pcd.pf9.io/grafana/login`
Reset Grafana admin password (advanced)
{% hint style="danger" %}
Resetting Grafana independently is not recommended. Keep Grafana and UI admin passwords in sync.
{% endhint %}
The namespace is `pcd` by default. If you customized the FQDN, use the FQDN shortname for the namespace. Example: If the FQDN is *ce.acmeretail.com*, assign `ce` to the `NS` variable in the below example.
```bash
NS=pcd
kubectl exec -it deploy/grafana -n $NS -- grafana cli admin reset-admin-password
```
Uninstall Community Edition
```bash
/opt/pf9/airctl/airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml
```
Expand an Ubuntu logical volume (LVM)
If your root filesystem is smaller than the physical volume, expand it.
1. Find the root device:
```bash
df -h /
```
2. Resize the LV (example name shown):
```bash
sudo lvresize -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
```
3. Resize the filesystem:
```bash
sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
```
### Troubleshooting
Start with [Common Issues](/private-cloud-director/getting-started/getting-started-with-community-edition/common-issues).
#### Optional learning resources
Platform9 YouTube:
* Channel:
* Playlist: [Private Cloud Director playlist](https://www.youtube.com/playlist?list=PLUqDmxY3RncUmegG6dv8XxjSr0g1THpLq)
# Custom Installation
The purpose of this page is to document the ways that Community Edition can be customized.
{% hint style="danger" %}
**Community Edition is not for production workloads.** Use it for labs, evaluation, and learning only.
{% endhint %}
There are a number of environment variables that can be defined prior to deploying the Community Edition host that will allow you to customize the default IP address used for the install, the URL for the Community Edition user interface, the use of install telemetry, and more.
{% hint style="info" %}
**Info**
All environment variables exported via the command line will not persist beyond the current terminal session, and only apply to the user that exports the variables. If a variable needs to be exported, run the install from the same terminal session. All environment variables will be exported using the following command structure: `export =`
{% endhint %}
### HTTP & HTTPS proxy support
If you would like to install Community Edition behind a proxy, you can use the `HTTPS_PROXY` and `HTTP_PROXY` environment variables. Additionally, the `NO_PROXY` environment variable can be used to add addresses that should not be proxied to the default list of addresses.
```
export HTTPS_PROXY=https://proxy:1234
export HTTP_PROXY=http://proxy:5678
export NO_PROXY=
```
### Change IP address
The install script will choose the IP address of the interface set as the default route using the `ip route` command. Should you choose to use a different default IP address, you can export the `IP_ADDRESS` environment variable.
```bash
export IP_ADDRESS=192.168.1.150
```
### Deployment FQDN (fully qualified domain name)
The FQDN for the deployment (`DU_FQDN`) sets the URL for the Community Edition user interface. It may be useful to customize the fully qualified domain name based on an existing DNS domain, for example.
Defaults:
* `DU_FQDN="pcd.pf9.io"`
* Resulting fully qualified domain name: `pcd.pf9.io`
```bash
export DU_FQDN="pf9.lan"
# This will result in pf9-test.lan as the FQDN for the user interface
```
### Installing with a signed SSL certificate
CE can be installed with a signed SSL certificate instead of the default self-signed certificate. You may do this by setting both `USER_CERT_PATH` and `USER_KEY_PATH` environment variables with the fully qualified paths to the certificate and key files.
It is required to generate the certificates with the appropriate wildcard SANs and Key Usage:
* \*.pf9.localnet
* \*.domain.net
The first, \*.pf9.localnet is required for internal usage. The second depends on the shortname/FQDN used. For example if the `DU_FQDN` is "air99.platform9.net", then ensure the certificate has SANs for \*.platform9.net.
In addition, ensure the following Key Usage extensions are enabled:
```none
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
```
Defaults:
* These variables are not set by default, as installing a self-signed certificate is the default behavior.
```bash
export USER_CERT_PATH=/path/to/cert
export USER_KEY_PATH=/path/to/key
```
### Community Edition deployment networking
Community Edition is installed as a Kubernetes deployment, and uses internal networking for communication between pods (`POD_CIDR` ) and for exposing applications as service IPs (`SERVICE_CIDR` ). If these IP address ranges conflict with your organization's networking ranges, they can be changed as needed. If there is no conflict, there's no need to change these default ranges.
Defaults:
* `SERVICE_CIDR="10.21.0.0/16"`
* `POD_CIDR="10.20.0.0/16"`
```bash
export SERVICE_CIDR="10.21.0.0/16"
export POD_CIDR="10.20.0.0/16"
```
### Installation telemetry
The Community Edition installation collects anonymous telemetry during the installation process allowing Platform9 to improve the installation process. Should you wish to opt out, the `TELEMETRY` environment variable can be set to `false`.
Default:
* `TELEMETRY=true`
```bash
export TELEMETRY=false
```
### Skip install pre-requisites check
The Community Edition installer runs a series of pre-requisite checks before installing, such as validating the amount of CPU & memory available. This behavior can be changed by setting the `SKIP_PRECHECKS` environment variable to `true` .
Default:
* `SKIP_PRECHECKS=false`
{% hint style="warning" %}
**Warning**
Forcing an install with less than the required amount of CPU & memory *will* cause it to fail!
{% endhint %}
```bash
export SKIP_PRECHECKS=true
```
### Skip post-install experience poll
As part of our ongoing effort to make the install experience as smooth & painless as possible, we've added a post-install poll to capture how you rated the installation experience. The `SKIP_RATING_PROMPT` variable can be set to `true` to disable this behavior.
Default:
* `SKIP_RATING_PROMPT=false`
```bash
export SKIP_RATING_PROMPT=true
```
### Automatically accept EULA
The `ACCEPT_EULA` environment variable can set to `true` before installing Community Edition, providing you have read and agreed to the EULA in advance. This environment variable can useful in the case of automating Community Edition installs.
Default:
* `ACCEPT_EULA=false`
```bash
export ACCEPT_EULA=true
```
### Automatically clean-up previous installation
The `SHOULD_CLEANUP` environment variable can be set to `true` to bypass the prompt to keep or delete a prior installation. The default is `false` .
Default:
* `SHOULD_CLEANUP=false`
```bash
export SHOULD_CLEANUP=true
```
### Extending default deployment timeout
If you are experiencing install failures due to timeouts, they may be resolved by setting the install script's default timeout period to longer than 600 seconds using the `SVC_DEPLOYMENT_TIMEOUT` environment variable. The normal installation process uses an internal variable, which can be changed with this environment variable.
Default:
* The default internal timeout period is `600` seconds.
```bash
export SVC_DEPLOYMENT_TIMEOUT=1200
```
### Kubernetes workload support
Kubernetes management plane support is optional in CE, enabled via the interactive install prompt or by pre-exporting `ENABLE_K8S` as true. The default is `false`.
```bash
export ENABLE_K8S=true
```
### Protected environment variables
The following environment variables are critical to the installation process and must not be changed, as this will cause installation failures: `S3_BUCKET` and `S3_USER_AGENT`.
# Common Issues
The purpose of this page is to provide solutions to common issues during Community Edition (CE) installation & operation.
{% hint style="danger" %}
**Community Edition is not for production workloads.** Use it for labs, evaluation, and learning only.
{% endhint %}
## Resolving failed prerequisites checks
### Ports should not be in use
The CE installer expects to be installed in a dedicated system with the following ports available: 443, 2379, 2380, 3306, 4194, 5395, 5672, 5673, 6264, 8023, 8158, 8285, 8558, 9080, 10250, 10255. Typically, this error will be encountered after a previous installation failure. To completely clean up an installation failure, use `airctl delete-cluster` and then retry the installation.
```bash
# Deletes the CE installation, including k3s
sudo airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml
# deletes logs & downloaded artifacts in the user's home directory
rm -r "$HOME/airctl-logs"
rm -r "$HOME/pcd-ce"
# runs the install script from the internet
curl -sfL https://go.pcd.run | bash
```
### Insufficient CPU or RAM
The CE install script checks the number of logical CPUs available at system start. If the install fails this check, run the installation again after provisioning more logical CPUs to the system (if running as a virtual machine) or on physical hardware with more logical CPUs available.
The install script calculates the amount of usable memory using the `MemTotal` line in `/proc/meminfo` , and checks for an amount less than 28GB. Additional memory must be made available to the CE system.
### Multiple default routes with the same metric
The CE install script checks for more than one default IPv4 route with the same metric. If more than one route with the same metric is found, this check will fail, as the system will not know which default gateway to use. To resolve this error, either change the metric on the additional default routes, or delete any additional default routes with the same metric if they aren't needed.
```bash
# list all IPv4 routes
ip -4 route show
# Change the metric on the additional default route(s)
sudo ip route change default via metric
# Delete the additional default route(s) if they are not needed
sudo ip route del default via
```
### Unsupported OS or version
The CE install script checks the operating system name and version. If this check fails, please try installing using a supported operating system and version as described in the CE [Pre-requisites](/private-cloud-director/getting-started/getting-started-with-community-edition/prerequisites#detailed-prerequisites).
### Kernel panic or vm panic on oom
The CE install script checks to make sure kernel & oom (out of memory) panics are configured correctly. `/proc/sys/kernel/panic` should be set to `10` and `/proc/sys/vm/panic_on_oom` should be set to `0` .
### Failed to check firewalld status or service not found
The CE install script checks `systemctl` to see if `firewalld.service` is inactive or not running, and will fail if the service is set to active. To resolve this, use the following commands.
```bash
# Stop the firewalld service
sudo systemctl stop firewalld
# Prevent it from starting on boot
sudo systemctl disable firewalld
```
### IPv6 is disabled
The CE install script checks `/proc/sys/net/ipv6/conf/all/disable_ipv6` to see if it is set to `0` . IPv6 should be enabled by default, but if it has been explicitly disabled, please enable IPv6 and reboot the CE system before trying the installation again.
***
## Resolving installation failures
### Calico CNI
CE installation can fail if it encounters issues with Calico CNI (container networking interface). Calico sets up networking within CE's Kubernetes pods and provides DNS resolution. Typically, these issues are either related to slowness when retrieving the container images from a remote image repository causing the install script to timeout while waiting, or are related to a previous installation not being fully cleaned up before an installation is attempted again. These issues can usually be resolved by fully cleaning up the installation attempt and trying again.
You may also consider setting the install's default timeout to an amount higher than 600 seconds. See the [Extending default deployment timeout](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation#extending-default-deployment-timeout) section of the [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation) page.
```bash
# Deletes the CE installation, including k3s
sudo airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml
# deletes logs & downloaded artifacts in the user's home directory
rm -r "$HOME/airctl-logs"
rm -r "$HOME/pcd-ce"
# runs the install script from the internet
curl -sfL https://go.pcd.run | bash
```
### Failed to apply logrotation on node - exit status 127
The CE installation will fail if it is unable to create a cronjob to automatically rotate log files, and will return exit status 127 if `cron` is not installed on the system. This is a default package on the Ubuntu 22.04 AMD64 server cloud image distribution, but is not included with the Ubuntu 22.04 AMD64 minimal cloud image. Installing `cron` will resolve this issue, but using the minimal cloud image is not supported and doing so may result in additional install failures.
### Curl responds with an error, or does not work
This is typically caused by network restrictions such as firewalls or lack of internet access. Using `curl` with verbose mode enabled (`-v` ) is useful for debugging.
```bash
# Example
curl -v https://go.pcd.run | bash
```
### Installation fails due to curl timeouts
This issue typically occurs early in the installation process, and can happen when the CIDRs used for Kubernetes services & pods conflicts with external local area networks. The default for the services CIDR is `10.21.0.0/16`, and the default for the pods CIDR is `10.20.0.0/16`. See the [Community Edition deployment networking](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation#community-edition-deployment-networking) section on the [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation) page for information on how to change these defaults using environment variables. After the environment variables are set, use the same terminal session to follow the steps in the section titled "Recovering a failed installation" on this page.
### Failed to find running pod percona-db-pxc-db-pxc-0
This error can occur when installing Community Edition on CPUs older than Intel Nehalem or AMD Bulldozer generations, as Percona requires the `x86-64-v2` microarchitecture level. This can be confirmed by running the following command:
```bash
sudo /lib64/ld-linux-x86-64.so.2 --help
```
Look for `Subdirectories of glibc-hwcaps` directories in the output. The `x86-64-v2` line should not say `supported, searched` .
Please try installing Community Edition again on CPUs newer than Intel Nehalem or AMD Bulldozer generations.
### Required ports & external URLs
Required ports:
* TCP: 53, 443
* UDP: 53, 123
Should you need to allow access to external URLs in order to complete a Community Edition installation, here are all of the URLs that are currently accessed during install:
```none
api2.amplitude.com
auth.docker.io
cdn.dl.k8s.io
cdn01.quay.io
check.percona.com
checkpoint-api.hashicorp.com
cr.fluentbit.io
dl.k8s.io
dockermirror.platform9.io
docs.tigera.io
gcr.io
ghcr.io
github.com
go.pcd.run
grafana.com
opencloud-dev-charts.s3.us-east-2.amazonaws.com
pcd-community.s3-accelerate.amazonaws.com
pkg-containers.githubusercontent.com
prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com
production.cloudflare.docker.com
pypi.org
quay.io
registry-1.docker.io
registry.k8s.io
release-assets.githubusercontent.com
storage.googleapis.com
us-east4-docker.pkg.dev
usage.projectcalico.org
```
Here is the list of required external URLs in order to onboard a hypervisor host to Community Edition.
```none
esm.ubuntu.com
files.pythonhosted.org
github.com
motd.ubuntu.com
pcdctl.s3.us-west-2.amazonaws.com
pypi.org
www.python.org
```
***
## Recovering a failed installation
Situation: A Community Edition installation has failed, the issue has been rectified, and you'd like to restart the installation.
Resolution: You must first uninstall/delete the CE installation (and k3s) using the following command, and then run the install script again.
```bash
# Fully delete CE installation & k3s
/opt/pf9/airctl/airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml
# Run install script again
curl -sfL https://go.pcd.run | bash
```
After these steps have completed successfully, you may complete the rest of the deployment process starting with the [Local DNS Entries](/private-cloud-director/getting-started/getting-started-with-community-edition#local-dns-entries) section in [Getting Started With Community Edition](/private-cloud-director/getting-started/getting-started-with-community-edition).
***
## General troubleshooting
For CE hosts, use the following commands to understand where the error is occurring:
* Check CE's status after installing to see which region (Kubernetes namespace) is experiencing issues. `/opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml`
* For troubleshooting a failed install, check the install logs at `airctl-logs/airctl.log`
* Check the size and amount of available filesystem space `df -h /`
* `kubectl describe node` will show information about CE's underlying Kubernetes infrastructure
* if resources are constrained, the allocated resources block should show requests CPU and memory near or at 100%
* `kubectl get pods -A` Look for any pods that are not Running or Completed. Pods in CrashLoopBackOff or with high restart counts and a young age should have their logs investigated for more detail.
* `kubectl logs -n ` to view pod logs.
* The default Kubernetes namespace is `pcd` for the infrastructure region, unless the `DU_FQDN` environment variable was customized prior to installation as described in the Deployment FQDN section of [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation).
# Backup and Restore CE
This document describes the backup & restore process for Private Cloud Director Community Edition (CE).
{% hint style="danger" %}
**Community Edition is not for production workloads.** Use it for labs, evaluation, and learning only.
{% endhint %}
## Community Edition backup & restore process
### Scope
This procedure backs up and restores the **Community Edition management plane** on the CE host.
It does not back up VM disks, external storage, or anything outside the CE host.
### Before you begin
* Run as a user with `sudo` access on the CE host.
* Ensure `/tmp` has enough free space for the backup file.
* Plan for downtime during restore. `unconfigure-du` stops the CE management plane.
{% hint style="warning" %}
The backup can include secrets (tokens, passwords, certificates). Store it securely and restrict file permissions.
{% endhint %}
The backup process for Community Edition is:
* Create a backup using `airctl backup`. The resulting backup file will be saved in the `/tmp` directory on the CE host.
* Save backup copies of the `k3s.yaml` and `airctl-config.yaml` configuration files as well.
* Move the backup file, and copy the other files to another directory to prevent unintentional overwrite.
The restore process for Community Edition is:
* `airctl unconfigure-du` will stop the Community Edition management plane, including all of its Kubernetes pods, but will leave the Kubernetes install in place.
* `airctl restore` uses the backup file to rebuild & restore your Community Edition installation.
The full command list is outlined below:
{% stepper %}
{% step %}
#### Create a backup (saves to /tmp)
```bash
/opt/pf9/airctl/airctl backup --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endstep %}
{% step %}
#### Create a backup directory, and move the backup file
```bash
mkdir backups
mv /tmp/backup_* backups
```
{% endstep %}
{% step %}
#### Save a copy of k3s and airctl configuration files
```bash
cp /opt/pf9/airctl/conf/airctl-config.yaml backups/
cp /etc/rancher/k3s/k3s.yaml backups/
```
{% endstep %}
{% step %}
#### (Recommended) Lock down backup permissions
```bash
chmod 700 backups
chmod 600 backups/*
```
{% endstep %}
{% step %}
#### Verify the backup files exist
```bash
ls -lh backups/
```
{% endstep %}
{% step %}
#### Unconfigure the CE deployment unit
```bash
/opt/pf9/airctl/airctl unconfigure-du --config /opt/pf9/airctl/conf/airctl-config.yaml --force
```
{% endstep %}
{% step %}
#### Restore the deployment unit from the backup
```bash
/opt/pf9/airctl/airctl restore --config /opt/pf9/airctl/conf/airctl-config.yaml --backupdir backups
```
{% endstep %}
{% step %}
#### Verify the restore
Your deployment is healthy when regions show `Ready` and ready services match desired services.
```bash
/opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endstep %}
{% endstepper %}
# Self Hosted
space.vars.product\_name can be deployed in your own infrastructure, giving you complete control over your cloud management platform while maintaining data sovereignty and security.
### Overview
The space.vars.self\_hosted\_product\_name deployment model allows you to run space.vars.product\_name entirely within your own data center or private cloud environment. This option is ideal for organizations with strict compliance requirements, air-gapped environments, or specific infrastructure needs.
### Getting Started
#### Prerequisites
Review the system requirements, network configuration, and infrastructure prerequisites needed to deploy space.vars.product\_acronym in your environment.
[View Prerequisites →](/private-cloud-director/getting-started/self-hosted/self-hosted-pre-requisites)
#### Installation
Follow our step-by-step installation guide to deploy space.vars.product\_acronym in your infrastructure. The installation process covers initial setup, configuration, and verification.
[Installation Guide →](/private-cloud-director/getting-started/self-hosted/self-hosted-install)
#### Upgrade
Learn how to upgrade your space.vars.self\_hosted\_product\_name deployment to the latest version while maintaining service availability and data integrity.
[Upgrade Guide →](/private-cloud-director/getting-started/self-hosted/upgrade)
For operational depth on host upgrades — including pre-upgrade environment checks, host role sequencing, Ubuntu 22.04-to-24.04 OS upgrade caveats, and failure recovery — see the [Host Upgrade Runbook](/private-cloud-director/upgrade/host-upgrade-runbook).
After completing the upgrade, follow the [Post-Upgrade Verification](/private-cloud-director/upgrade/post-upgrade-verification) checklist to confirm all services and roles are healthy before re-enabling VM HA and DRR.
# Pre-requisites
This document outlines the prerequisites for deploying the space.vars.self\_hosted\_product\_name.
## Management Cluster
As part of the installation process, the space.vars.self\_hosted\_product\_name creates a **K3s cluster** using the physical servers that you use to deploy it on. We refer to this cluster as the **management cluster**. The space.vars.product\_name management plane then runs as a set of Kubernetes pods and services on this management cluster.
Single-node deployments are currently not supported for space.vars.self\_hosted\_product\_name. The minimum supported configuration requires 3 servers to ensure high availability and proper operation of the K3s management cluster. The 3-node cluster also lets core control-plane components — including the RabbitMQ message broker — run as multi-replica HA clusters by default, so a single management-server failure does not interrupt the management plane. For development or testing purposes, contact your Platform9 support for alternative deployment options.
The following is the recommended capacity for the management cluster, based on the projected scale of your space.vars.product\_name deployment. These configurations assume production deployments with high availability requirements.
| Hypervisors You Plan to Use | Minimum Capacity | Recommended Capacity |
| ------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
Small
(<20 hosts)
|
3 servers, each with:
20 vCPUs, 40GB RAM and 200GB SSD
|
3 servers, each with:
24 vCPUs, 48GB RAM and 250GB SSD
|
|
Growth
(<100 hosts)
|
3 servers, each with:
26 vCPUs, 48GB RAM and 250GB SSD
|
Option A — 3 servers, each with: 32 vCPUs, 64GB RAM and 250GB SSD
Option B — 5 servers, each with: 24 vCPUs, 48GB RAM and 250GB SSD
|
|
Enterprise
(>100 hosts)
|
3 servers, each with:
32 vCPUs, 96GB RAM and 250GB SSD
|
Option A — 3 servers, each with: 48 vCPUs, 128GB RAM and 250GB SSD
Option B — 5 servers, each with: 32 vCPUs, 80GB RAM and 250GB SSD
|
The above recommendation is for a single region. ***For each additional region*** deployed on the same management cluster, use the same per-tier table to size the incremental capacity, based on the number of hosts that region will manage. It is recommended to have a separate management cluster in each geographical location, to avoid performance degradation and a single point of failure.
For Growth and Enterprise scale, two recommended options are provided. The 3-server option keeps the cluster footprint small and is suitable when minimizing the number of management servers is a priority. The 5-server option distributes the core services and their replicas across more nodes, providing additional headroom to absorb a node failure, while lowering the per-node resource requirement. The total cluster capacity is comparable in both options.
### Disk Partition Guidance
space.vars.product\_name normally runs on a single root filesystem. If your environment requires dedicated partitions for compliance or security, size the directories that space.vars.product\_name depends on: **/var**, **/opt**, and **/etc**. These directories hold logs, container data, PF9 components, and configuration files, so they must have enough space to support normal operations and upgrades.
Recommended Sizes for /var, /opt, and /etc
| Directory | Minimum | Recommended |
| --------- | ------- | ----------- |
| /var | 130GB | 150GB |
| /opt | 10GB | 20GB |
| /etc | 1GB | 2GB |
| /tmp | 15GB | 20GB |
| /home | 15GB | 20GB |
| /root | 15GB | 25GB |
### Server Configuration
Each physical server that you use to run as part of the management cluster should meet the following requirements:
**Operating System**: Ubuntu 22.04, Ubuntu 24.04
### **Swap config**
Make sure that each server has swap disabled. You can run the following command to do this.
{% tabs %}
{% tab title="Bash" %}
```bash
swapoff -a
```
{% endtab %}
{% endtabs %}
The above change will not survive a reboot; hence, it is recommended to update the `/etc/fstab` file and comment out the line that has the entry for the `swap` partition. e.g.
{% tabs %}
{% tab title="Bash" %}
```bash
UUID=aabbcc / ext4 errors=remount-ro 0 1
UUID=xxyyzz /home ext4 defaults 0 2
UUID=mswmsw /media/windows ntfs defaults 0 0
#/dev/sdb1 none swap sw 0 0 <--- comment out the line
```
{% endtab %}
{% endtabs %}
### **IPv6 support**
Ensure the below sysctl setting is set to 0, so that IPv6 support is enabled on the server.
{% tabs %}
{% tab title="Bash" %}
```bash
sysctl net.ipv6.conf.all.disable_ipv6
# If currently set to 1, change it to 0 as below:
echo net.ipv6.conf.all.disable_ipv6=0 >> /etc/sysctl.conf
sysctl -p
```
{% endtab %}
{% endtabs %}
### **Passwordless Sudo**
Many operations require sudo access (for example, installing Yum repositories, Docker, etc.). Please ensure that your server has passwordless sudo enabled.
**Kernel Panic Option**
Update the server configuration section to include a step for setting `kernel.panic=10`
{% tabs %}
{% tab title="YAML" %}
```yaml
echo "kernel.panic=10" >> /etc/sysctl.conf && sysctl -p
```
{% endtab %}
{% endtabs %}
### **SSH Keys**
* We rely on SSH to log in to the management cluster hosts and to install various components and manage them.
* Please generate ssh keys and sync them across all hosts of the management cluster. The airctl installation is performed using a ***non-root user,*** so generate and use the SSH keys with the same non-root user that you select to run airctl operations. This ensures SSH access between hosts is established under the same user that performs the airctl operations.
* We recommend generating the key pair on one host and then adding the public key to all other hosts in their `~/.ssh/authorized_keys` file. This will enable every host in the management cluster to ssh into every other host.
{% tabs %}
{% tab title="Bash" %}
```bash
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh-copy-id -i ~/.ssh/id_rsa.pub root@test-3
```
{% endtab %}
{% endtabs %}
### **Package Updates**
* Install `cgroup-tools` :
{% tabs %}
{% tab title="Bash" %}
```bash
apt-get update -y && apt-get install cgroup-tools -y && apt-get install iptables -y
```
{% endtab %}
{% endtabs %}
* Download and Update OpenSSL Version to 3.0.7 for Ubuntu 22.04:
{% tabs %}
{% tab title="Bash" %}
```bash
export AGENT_KEY=
# Download the OpenSSL package
curl --user-agent "${AGENT_KEY}" https://pf9-airctl.s3-accelerate.amazonaws.com/openssl-smcp-ubuntu/openssl_3.0.7-1_amd64.deb --output /tmp/openssl_3.0.7-1_amd64.deb
# Verify the MD5 checksum
md5sum /tmp/openssl_3.0.7-1_amd64.deb | grep 706caf || { echo "MD5 checksum does not match, exiting." && exit 1; }
# Install the OpenSSL package
sudo dpkg -i /tmp/openssl_3.0.7-1_amd64.deb || { echo "Failed to install OpenSSL, exiting." && exit 1; }
echo "/usr/local/ssl/lib64" | sudo tee /etc/ld.so.conf.d/openssl-3.0.7.conf
sudo ldconfig -v
# Create a symbolic link to the OpenSSL binary
sudo ln -sf /usr/local/ssl/bin/openssl /usr/bin/openssl
# Verify the OpenSSL version
openssl version | grep 3.0.7 || { echo "OpenSSL version does not match, exiting." && exit 1; }
```
{% endtab %}
{% endtabs %}
* User Agent Key For Installation
You will need a specific Platform9 user agent key for the installation of your self-hosted management plane. Your Platform9 sales engineer will share the key with you prior to the install.
## Networking
You will need 2 virtual IPs that are on the same L2 domain as the hosts in the management cluster.
* VIP #1: This is the IP where you can access the space.vars.product\_name management plane UI.
* VIP #2: This is used to serve the management K3s cluster's API server.
## Storage
For a production setup of space.vars.self\_hosted\_product\_name you will need a Kubernetes Container Storage Interface (CSI) compatible storage for persisting the state of the management cluster. To know more, see [CSI and Kubernetes Storage](https://kubernetes.io/docs/concepts/storage/volumes/).
The Terrakube component of PCD AppCatalog requires persistent storage with multiple access (RWX) for sharing among Terrakube pods. Ensure a compatible RWX storage solution (like NFS, CephFS, or any CSI-compliant RWX provider) is available and configured in the K3s cluster where Terrakube runs.
### Storage Class Customisation
Customers have the flexibility to utilize a custom storage provisioner, allowing them to modify the *storage class* and *disk size* for each component according to their specific deployment needs. This section is optional; feel free to skip it if the default configuration meets your requirements.
**Default Storage Classes**
| Component | Default Storage Class | Comment |
| --------------- | -------------------------------- | ------------------------ |
| RabbitMQ | `pcd-sc` | NFS type storage |
| MySQL | Cluster default (`hostpath-csi`) | |
| OVN OVSDB NB | `pcd-sc` | NFS type storage |
| OVN OVSDB SB | `pcd-sc` | NFS type storage |
| Prometheus | `pcd-sc` | NFS type storage |
| Terrakube | `pcd-sc` | Requires RWX access mode |
| Terrakube Redis | Cluster default (`hostpath-csi`) | |
| Grafana | Cluster default (`hostpath-csi`) | |
| Audit | `pcd-sc` | |
To use the custom storage class provisioner, please follow these steps:
1. Place your custom storage class YAML files in the directory located at `/opt/pf9/airctl/conf/ddu/storage/custom/` . It is essential to have a storage class named `pcd-sc` included along with any other YAML files in this custom path.
2. Utilize `-p` or `--storage custom` in the `airctl configure` command. This will automatically apply all the YAML files found in the custom path.
*Example storage class yaml files:*
{% tabs %}
{% tab title="NFS Storage Class Example" %}
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name:
mountOptions:
- nfsvers=4.1
- nolock
parameters:
server:
share:
provisioner: nfs.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="Hostpath Provisioner Storage Class Example" %}
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name:
parameters:
storagePool: standard
provisioner: kubevirt.io.hostpath-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
```
{% endtab %}
{% endtabs %}
Override storage class and disk size for specific component:
Edit the `/opt/pf9/airctl/conf/options.json` file to configure the settings specifically related to storage parameters. This adjustment empowers you to customize the storage class to meet your precise requirements. Updating the disk size is optional, as default sizes are already set.
Ensure that the storage class overrides specified in `options.json` are available in the custom storage path. If the storage class is not present, the deployment will fail.
{% tabs %}
{% tab title="Options.json" %}
```json
# Values here are given for example and not actual values.
{
"chart_url": "",
"terrakube_sc": "efs-sc",
"terrakuberedis_sc": "block-sc",
"rabbitmq_sc": "block-sc",
"mysql_sc": "block-sc",
"ovn_ovsdb_nb_sc": "efs-sc",
"ovn_ovsdb_sb_sc": "efs-sc",
"grafana_sc": "block-sc",
"prometheusopenstack_sc": "efs-sc",
"audit_pvc_sc": "audit-sc",
"terrakube_disk_size": "11Gi",
"terrakuberedis_disk_size": "2048Mi",
"rabbitmq_disk_size": "6144Mi",
"mysql_disk_size": "10Gi",
"ovn_ovsdb_nb_disk_size": "11Gi",
"ovn_ovsdb_sb_disk_size": "12Gi",
"grafana_disk_size": "9Gi",
"prometheusopenstack_disk_size": "9Gi",
"audit_pvc_size": "6Gi"
}
```
{% endtab %}
{% endtabs %}
**Note:**
* Disk size units can be specified as Mi, Gi, or Ti.
* Decreasing disk size is **not supported**.
* Changing storage class name is supported only during installation, **not during upgrade.**
### **Inotify and Kernel limit Tuning**
Increase inotify limits to prevent airctl install from failing due to inotify errors
```bash
echo 'fs.inotify.max_queued_events = 512000
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 524288
fs.aio-max-nr = 500000
vm.panic_on_oom = 0' | sudo tee /etc/sysctl.d/99-pf9-airctl.conf > /dev/null
sudo sysctl -p /etc/sysctl.d/99-pf9-airctl.conf
```
# Install
This guide outlines the steps for a self-hosted deployment of space.vars.product\_name. Before installing, refer to the [Pre-requisites](/private-cloud-director/getting-started/pre-requisites) section to ensure all required prerequisites are met.
## Concepts
**Management Cluster**
As part of the installation process, the Self-Hosted version of space.vars.product\_name creates a K3s cluster using the physical servers that you use to deploy it on. We refer to this cluster as the **management cluster**. The space.vars.product\_name **management plane** then runs as a set of Kubernetes pods and services on this management cluster. The RabbitMQ message broker that mediates inter-service communication is deployed as a 3-node cluster on the management cluster by default for high availability.
**Infra Region vs Workload Regions**
Read [Tenant](/private-cloud-director/identity-and-multi-tenancy/tenant) to understand the concepts of regions and `infra` region in space.vars.product\_name.
## Download Installer
`airctl` is the command-line installer for space.vars.self\_hosted\_product\_name. Run the following commands only on one of the management cluster hosts to download `airctl` along with the required installer artifacts.
All the following steps should be performed by a non-root user.
#### Step 1: Download the Installer Script
Run the following command to download the installer script and required artifacts into your home folder:
{% tabs %}
{% tab title="Bash" %}
```bash
curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/latest/index.txt | awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/latest/" $NF "\" -o ${HOME}/" $NF}' | bash
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**NOTE**
Replace `YOUR_USER_AGENT_KEY` in the command with the user agent key you requested from Platform9. For more details see [Pre-requisites](/private-cloud-director/getting-started/pre-requisites)
{% endhint %}
#### Step 2: Make the Installer Executable
Set the execute permissions on the installation script.
{% tabs %}
{% tab title="Bash" %}
```bash
chmod +x ./install-pcd.sh
```
{% endtab %}
{% endtabs %}
#### Step 3: Run the Installation Script
Execute the installer with the specified version. This runs the installer using the version number found in `version.txt`.
{% tabs %}
{% tab title="Bash" %}
```bash
./install-pcd.sh `cat version.txt`
```
{% endtab %}
{% endtabs %}
#### Step 4: Add airctl to System Path
Add `airctl` to the system path to use it globally by creating a symlink in `/usr/bin` folder.
{% tabs %}
{% tab title="Bash" %}
```bash
sudo ln -s /opt/pf9/airctl/airctl /usr/bin/airctl
```
{% endtab %}
{% endtabs %}
### Configure airctl
Run the following command to generate a configuration file, which will be used to deploy the space.vars.self\_hosted\_product\_name management cluster.
{% tabs %}
{% tab title="Bash" %}
```bash
/opt/pf9/airctl/airctl configure --du-fqdn pcd.platform9.localnet --external-ip4 10.149.106.15 --ipv4-enabled --master-ips 10.149.106.11,10.149.106.12,10.149.106.13 --master-vip-interface ens3 --master-vip4 10.149.106.16 --storage-provider custom --regions Region1 --worker-ips 10.149.106.14 --node-az-labels /path/to/node-az-labels.yaml
```
{% endtab %}
{% endtabs %}
Following are the input parameters for the command:
1. **du-fqdn -** Specify the base FQDN you would like to use for product\_name. For eg `pcd.mycompany.com`
2. **external-ip4 -** Specify the VIP to be used for the management plane here.
3. **ipv4-enabled** - Enable the IPv4 networking for the cluster.
4. **master-ips** *-* Specify a comma separated list of IP addresses for the master nodes you'd like to use for the management cluster. We recommend minimum 3 master nodes for a production environment. These nodes act as worker nodes as well.
5. **worker-ips (optional)** - If you'd like to add worker nodes to the management cluster, then specify the comma separated list of IP addresses for worker nodes here.
6. **master-vip-interface** - Specify the name of the network interface to be used for the Virtual IP of master nodes. Note that each master node must have it's default network interface named with this name.
7. **master-vip4** - Specify the Virtual IP to be used for the management cluster. This will be used to serve the management Kubernetes cluster's API server.
8. **master-vip-vrouterid** - This is optional. If unspecified, one will randomly be generated and can be found in the updated cluster spec saved in the directory that contains the state of the management server (see 'Important Directories' section for directory location). It is recommended to specify one if you plan to deploy multiple Kubernetes clusters in the same VLAN to avoid collision.
9. **regions** - Specify one or more region names as a space-separated list for regions you would like to create in your space.vars.product\_name setup. When specifying more than one regions the list needs to be enclosed in "". The final FQDN for your space.vars.product\_name deployment will use a combination of your base FQDN and your region name. For example, if your base FQDN is `pcd.mycompany.com` and you specified a single region name here as region1, the final FQDN for your deployment will be `pcd-region1.mycompany.com`.
10. **storage-provider** - Specify the CSI storage provider that should be used to store any persistent state for the management cluster. If not specified, `hostpath-provisioner` storage provider option will be selected as default.
1. For `custom` as the storage provider, copy provider specific CSI yaml files to `/opt/pf9/airctl/conf/ddu/storage/custom/` locally. `airctl` will configure the dependencies reading the yaml files at this path and use the provided storage class for provisioning volumes for space.vars.product\_name components. There must be a yaml containing a `StorageClass` named `pcd-sc` and marked as the default StorageClass via the annotation `storageclass.kubernetes.io/is-default-class: "true"`.
2. Alternatively, you can create the storage provider resources and storage class out of band prior to running `airctl start` on the management cluster. **Ensure that the storage class is named `pcd-sc` and is marked as default** (via `storageclass.kubernetes.io/is-default-class: "true"`) in this case. We also recommend creating a test pod with a persistent volume to ensure correct connectivity and configuration is in place first.
3. For non-production environments and where custom storage provider is not available/required, select `hostpath-provisioner`.
11. **nfs-ip,** **nfs-share -** If using hostpath-provisioner as storage provider, NFS server IP and mount location must be provided.
1. You should have an NFS server pre-configured before selecting this option.
12. **node-az-lables (optional)** - Provide labels and distribute the workload equally ([detailed doc](https://docs.platform9.com/private-cloud-director/~/revisions/Vf3rc8gZIkCBQty9ryKP/getting-started/self-hosted/availability-zones))
13. **enable-k8s (optional)** - Install the Kubernetes management plane alongside the space.vars.product\_name management plane, which lets you provision managed Kubernetes clusters. This is opt-in; omit the flag to leave it disabled.
{% hint style="info" %}
**Kubernetes management plane**
The Kubernetes management plane is optional and uses additional resources on the management plane. If you skip it during installation, you can enable it later on the existing deployment. See [Enable Kubernetes Management Plane](/private-cloud-director/getting-started/self-hosted/airctl/enable-kubernetes-management-plane).
{% endhint %}
This command generates two configuration file templates:
* `/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml` – Contains the configuration required to bootstrap the management cluster.
* `/opt/pf9/airctl/conf/airctl-config.yaml` – Contains the configuration for the management plane.
To avoid passing the configuration file as a command-line option when running `airctl` commands, copy `/opt/pf9/airctl/conf/airctl-config.yaml` to your `$HOME` directory.
{% tabs %}
{% tab title="Bash" %}
```bash
ln -s /opt/pf9/airctl/conf/airctl-config.yaml $HOME/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
### Proxy Configuration (Optional)
If your environment uses a network proxy, set the required values in the `/opt/pf9/airctl/conf/helm_values/bork.template.yml` file as shown below:
{% tabs %}
{% tab title="Bash" %}
```bash
cat /opt/pf9/airctl/conf/helm_values/bork.template.yml | grep proxy
# Sample output:
https_proxy: "http://squid.platform9.horse:3128"
http_proxy: "http://squid.platform9.horse:3128"
no_proxy: "10.149.106.11,10.149.106.12,10.149.106.13,10.149.106.14,10.149.106.15,10.149.106.16,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.platform9.localnet,.default.svc"
```
{% endtab %}
{% endtabs %}
The list of I.P. addresses in the no\_proxy list should include the master-ips, worker-ips, external-ip4, master-vip4 along with any other addresses for which the traffic should not be routed via proxy server.
Also, to ensure that containerd honors the proxy values and allows the creation of the space.vars.product\_name management cluster, update the proxy values on all management plane nodes as shown below:
{% tabs %}
{% tab title="Bash" %}
```bash
cat /etc/environment
# Sample output:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/ssl/bin"
HTTP_PROXY="http://squid.platform9.horse:3128"
https_proxy="http://squid.platform9.horse:3128"
http_proxy="http://squid.platform9.horse:3128"
HTTPS_PROXY="http://squid.platform9.horse:3128"
NO_PROXY="10.149.106.11,10.149.106.12,10.149.106.13,10.149.106.14,10.149.106.15,10.149.106.16,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.platform9.localnet,.default.svc"
no_proxy="10.149.106.11,10.149.106.12,10.149.106.13,10.149.106.14,10.149.106.15,10.149.106.16,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.platform9.localnet,.default.svc"
```
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="Bash" %}
```bash
cat /etc/systemd/system/containerd.service.d/http-proxy.conf
# Sample output:
[Service]
EnvironmentFile=/etc/environment
```
{% endtab %}
{% endtabs %}
## Deploy Management Cluster
Next, deploy the management cluster for your space.vars.product\_name environment.
#### Step 1: Run Pre-Checks
Before creating the management cluster, run the following command to perform pre-checks and resolve any issues identified:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl check
```
{% endtab %}
{% endtabs %}
Optionally, configure AWS Credentials for `airctl` backup
Before creating the cluster, ensure that the AWS credentials required for S3 backup are available by creating the following file.
{% tabs %}
{% tab title="Bash" %}
```bash
Path: /etc/default/airctl-backup
Contents:
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
AWS_S3_PATH=s3://
```
{% endtab %}
{% endtabs %}
When you create this file before cluster deployment, the system automatically creates a Kubernetes secret named `aws-credentials` in the `pf9-utils` namespace. You need this secret to upload backups to your S3 bucket.
Without this file, you must manually create or patch the `aws-credentials` secret in the `pf9-utils` namespace after cluster creation.
#### Step 2: Deploy the K3s Cluster
Run the following command to deploy the K3s cluster:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl create-cluster --verbose
```
{% endtab %}
{% endtabs %}
#### Step 3: Validate Cluster Health
Once the cluster is created, verify that it is functioning properly and that all nodes are healthy:
{% tabs %}
{% tab title="Bash" %}
```bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get nodes
# Sample output:
NAME STATUS ROLES AGE VERSION
10.10.11.111 Ready control-plane,etcd,master 4d22h v1.33.9+k3s1
10.10.12.242 Ready control-plane,etcd,master 4d22h v1.33.9+k3s1
10.10.13.253 Ready control-plane,etcd,master 4d22h v1.33.9+k3s1
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**Info**
Please refer to `/var/log/k3s.log` for troubleshooting any issues with the management cluster creation.
{% endhint %}
***
## Install Management Plane
Now that the management cluster is created, run the following commands to install and configure the space.vars.product\_name self-hosted management plane.
{% tabs %}
{% tab title="Bash" %}
```bash
airctl start
# Sample output:
INFO pcd-virt management plane creation started
SUCCESS generating certs and config...
SUCCESS setting up base infrastructure...
▀ starting consul...Secret consul-gossip-encryption-key in namespace default not found, creating new...
INFO bork setup done, creating management plane
INFO starting pcd-virt deployment...
SUCCESS pcd-virt deployment now complete
INFO pcd-virt management plane created - the services will take a while to start
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**Info**
It may take up to 45 minutes for all services to be deployed.
{% endhint %}
### Monitor Deployment Progress
You can track the progress of the management plane deployment by checking the logs of the `du-install` pod as a `root` user:
{% tabs %}
{% tab title="Bash" %}
```bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get pod -A | grep du-install
kubectl logs -n -f
```
{% endtab %}
{% endtabs %}
Alternatively, to check specific pods:
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl get pods -n foo-bork | grep du-install
# Sample output:
du-install-foo-bmqqd 0/1 Completed 0 108m
du-install-foo-region1-f7fdw 0/1 Completed 0 98m
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**Info**
Please refer to `airctl-logs/airctl.log` for logs in case of any issues with `airctl` start command.
{% endhint %}
### Check Management Plane & Region Status
To verify the status of the management plane and its regions, run:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl status
```
{% endtab %}
{% endtabs %}
## Obtain UI Credentials
Once the installation is complete, you can retrieve the credentials for the space.vars.product\_name UI by running the following command:.
{% tabs %}
{% tab title="Bash" %}
```bash
airctl get-creds
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**DNS Considerations**
If you do not have a working internal DNS that resolves the management plane FQDN to its IP address, you must manually update the `/etc/hosts` file on your local machine and any new host you want to add to the management cluster.
{% endhint %}
Use the following command to check if the necessary entry exists:
{% tabs %}
{% tab title="Bash" %}
```bash
cat /etc/hosts | grep foo
foo-region1.bar.io
foo.bar.io
foo-bork.bar.io
```
{% endtab %}
{% endtabs %}
After updating the hosts file, open your web browser and log into the UI using the provided credentials. Then, follow the steps in the [Getting Started](/private-cloud-director/getting-started/getting-started) guide to configure your hosts and set up your space.vars.product\_name environment.
## Important Files & Directories
The following directories contain various log and state files related to the space.vars.product\_name self-hosted deployment.
#### Directories\*\*:\*\*
* `/opt/pf9/airctl` – Contains all binaries, offline installers, Docker image tar files, miscellaneous scripts, and configuration files for airctl.
* `/opt/pf9/pf9-kube` – Managed by Nodelet. Stores binaries and scripts used to manage the management cluster.
* `/etc/rancher/k3s/` – Contains configuration files for K3s and custom registries used by it.
* `/var/lib/rancher/k3s/` — stores K3s runtime data including certificates, containerd data, cluster state, manifests, and other internal components required for the cluster to run.
* `~/airctl-logs/` – Stores all logs related to the deployment.
#### Files\*\*:\*\*
* `/opt/pf9/airctl/conf/airctl-config.yaml` – Contains configuration information for the management plane.
* `/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml` – Stores configuration required to bootstrap the management cluster.
* `/var/log/k3s.log` – Log file for troubleshooting issues with management cluster creation.
* `~/airctl-logs/airctl.log` – Log file for troubleshooting issues with management plane creation.
# Install using the helper script
Deploy a self-hosted Private Cloud Director on-premises using the go.pcd.run/onprem helper script.
## Overview
The `go.pcd.run/onprem` helper script automates the on-premises deployment of self-hosted Private Cloud Director. It handles environment preparation tasks that the [Install](/private-cloud-director/getting-started/self-hosted/self-hosted-install) requires you to complete separately, including OpenSSL installation, swap configuration, kernel settings, and package dependencies, so you can deploy with fewer steps.
Use this method if you want a faster path to a running management plane on a network-connected environment. For air-gapped deployments, see [Air Gapped Installation](/private-cloud-director/getting-started/self-hosted/self-hosted-airgap-install).
### Prerequisites
Before running the script, ensure your environment meets the following requirements. Most prerequisites from the [Pre-requisites](/private-cloud-director/getting-started/self-hosted/self-hosted-pre-requisites) are handled automatically by the script. Only the following are required upfront:
* **Passwordless sudo**: The user running the script must have passwordless sudo configured.
* **SSH keys**: SSH key-based access must be set up so that the host running the script can connect to all master and worker nodes without a password prompt.
You also need a Platform9 user agent key. Your Platform9 sales engineer provides this key before installation.
#### Step 1: Set environment variables
The script requires a set of environment variables to run. Export the required variables and any optional ones that apply to your environment.
Replace all values in `<>` with your environment-specific values.
```bash
# Required
export USER_AGENT_KEY=""
# Optional — defaults shown
export SSH_KEY_PATH="" # Default: "${HOME}/.ssh/id_rsa"
export SSH_USER="" # Default: current user
export PCD_VERSION="" # Default: "latest"
# Optional — proxy settings
export HTTPS_PROXY=""
export HTTP_PROXY=""
export NO_PROXY=""
# Optional — custom storage and options overrides
export AIRCTL_CUSTOM_CSI_FILES_PATH="" # Required for custom storage class
export AIRCTL_OPTIONS_OVERRIDES_FILE_PATH="" # Required for options.json overrides
```
#### Step 2: Deploy space.vars.product\_acronym
Choose one of the following methods. Both methods require the environment variables from Step 1 to be exported first.
**Method 1: Environment variable-based installation**
Export your deployment parameters as environment variables, then run the script. Replace all values in `<>` with your environment-specific values.
```bash
export AIRCTL_MASTER_IPS=",," && \
export AIRCTL_MASTER_VIP4= && \
export AIRCTL_REGIONS="" && \
export AIRCTL_DU_FQDN= && \
export AIRCTL_EXTERNAL_IP4= && \
export AIRCTL_STORAGE_PROVIDER="hostpath-provisioner" && \
export AIRCTL_NFS_IP= && \
export AIRCTL_NFS_SHARE="" && \
export AIRCTL_IPV4_ENABLED=true && \
export AIRCTL_MASTER_VIP_INTERFACE=
```
To derive the full list of available environment variables, refer to the `airctl configure` flag list in the [Airctl Reference](/private-cloud-director/getting-started/self-hosted/airctl). Each long flag maps to an environment variable by prepending `AIRCTL_`, converting to uppercase, and replacing hyphens with underscores. For example, `--du-fqdn` becomes `AIRCTL_DU_FQDN`.
After exporting your variables, run the script:
```bash
curl -sS https://go.pcd.run/onprem | bash
```
**Method 2: Flag-based installation**
Pass deployment parameters directly to the script as flags. Replace all values in `<>` with your environment-specific values.
```bash
curl -sS https://go.pcd.run/onprem | bash -s -- \
--du-fqdn \
--external-ip4 \
--ipv4-enabled \
--master-ips ,, \
--master-vip-interface \
--master-vip4 \
--nfs-ip \
--nfs-share \
--storage-provider hostpath-provisioner \
--regions ""
```
For the full list of supported flags, see [Airctl Reference](/private-cloud-director/getting-started/self-hosted/airctl#configuration-commands) in the Airctl Reference.
{% hint style="info" %}
**NOTE**
The basic environment variables from Step 1, including `USER_AGENT_KEY` are required regardless of which method you use
{% endhint %}
### Proxy configuration (optional)
If your environment routes traffic through a network proxy, export the proxy variables before running the script. Replace all values in `<>` with your environment-specific values.
The `NO_PROXY` list must include all master node IPs, worker node IPs, `EXTERNAL_IP4`, and `MASTER_VIP4`, along with any other addresses that should bypass the proxy.
```bash
export HTTPS_PROXY="http://:"
export HTTP_PROXY="http://:"
export NO_PROXY=",,,,,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.default.svc"
```
### Custom CSI storage backend (optional)
By default, the script uses `hostpath-provisioner` as the storage provider. To use a custom CSI storage backend instead, complete the following steps before running the script.
1. Create a directory and place your custom `StorageClass` YAML files in it.
2. Export the path to that directory:
```bash
export AIRCTL_CUSTOM_CSI_FILES_PATH=""
```
3. Set the storage provider to `custom` using the appropriate variable or flag for your chosen method:
* Environment variable: `AIRCTL_STORAGE_PROVIDER="custom"`
* Flag: `--storage-provider custom`
{% hint style="warning" %}
**NOTE**
Your custom YAML files must include at least one storage class named `pcd-sc`, along with all storage classes referenced in `options.json` if you are using an overrides file.
{% endhint %}
### Overriding default options.json (optional)
To customize storage classes per component or adjust management plane deployment settings, you can override fields in the default `options.json`.
1. Create a JSON file containing only the fields you want to add or override.
2. Export the path to that file before running the script:
```bash
export AIRCTL_OPTIONS_OVERRIDES_FILE_PATH=""
```
{% hint style="info" %}
**NOTE**
This variable cannot be passed as a flag to the script. It must be exported as an environment variable.
{% endhint %}
# Air Gapped Installation
Deploy Platform9 Private Cloud Director (PCD) in an environment without direct internet access.
### Overview
Use this guide to deploy Platform9 space.vars.product\_name (space.vars.product\_acronym) in an air-gapped environment, where cluster nodes have no direct internet access. Before you start the installation, download all required packages, container images, and installation artifacts on a host with internet access, then transfer them into your environment.
In an air gapped deployment, the following components must be set up inside your environment before space.vars.product\_acronym can be installed. You will configure each of these as part of this guide.
| Component | Purpose |
| ----------------------------- | ------------------------------------------ |
| Private APT repository | Hosts required Ubuntu packages |
| Private container registry | Hosts required container images |
| Airctl installation artifacts | Installer components downloaded externally |
| NTP server | Ensures time synchronization across nodes |
### Prerequisites
Before you begin, ensure the following conditions are met.
#### Network requirements
All cluster nodes must have network connectivity to the following internal services:
* Private APT repository
* Private container registry
* NTP server
For management plane host configuration, follow the prerequisites guide, except for package updates and OpenSSL installation. Those steps are covered in this guide.
#### DNS requirements
Configure DNS resolution for the container registry hostname.
**Example:**
```
registry.pf9.io → 10.10.13.145
```
If DNS is not available, add the registry hostname to `/etc/hosts` on every node instead.
**Example:**
```
10.10.13.145 registry.pf9.io
```
### Step 1: Configure NTP synchronization
All nodes must synchronize time with an NTP server inside your network to prevent clock skew across the cluster. In this step, you configure each node to point to your internal NTP server, restart the time synchronization service, and verify that synchronization is active before proceeding. If NTP is already configured on your nodes, skip this step.
1. Configure the NTP client on each node to point at your internal NTP server.
```bash
sudo mkdir -p /etc/systemd/timesyncd.conf.d
echo "[Time]
NTP=" | sudo tee /etc/systemd/timesyncd.conf.d/custom.conf
```
2. Restart the time synchronization service and enable it to start on boot.
bash
```bash
sudo systemctl restart systemd-timesyncd
sudo systemctl enable systemd-timesyncd
```
3. Verify that the node is actively syncing with the NTP server before moving on.
bash
```bash
timedatectl status
timedatectl show-timesync --all
```
### Step 2: Set up a private APT repository
The private APT repository hosts the Ubuntu packages required by the platform. You create this repository on a server with internet access and then make it accessible to the air-gapped cluster. To complete this step, you will download the required scripts and dependency list, download the package dependencies, and then initialize the repository.
1. On a host with internet connectivity, download the repository scripts and the dependency list.
```bash
# Script to create a repo
curl --user-agent "" -O https://pf9-airctl.s3.us-west-1.amazonaws.com/latest/sample_scripts/create_apt_repo.sh
# Script to download package dependencies
curl --user-agent "" -O https://pf9-airctl.s3.us-west-1.amazonaws.com/latest/sample_scripts/download_all_deps.sh
# Dependency list
curl --user-agent "" -O https://pf9-airctl.s3.us-west-1.amazonaws.com/latest/dependency_list.txt
```
2. Make both scripts executable before running them.
```bash
chmod +x create_apt_repo.sh download_all_deps.sh
```
3. Download all package dependencies required for the installation.
```bash
./download_all_deps.sh
```
{% hint style="info" %}
**NOTE**
If you are using HTTPS with a custom CA, install the CA into `/usr/local/share/ca-certificates` and run `update-ca-certificates` before proceeding.
{% endhint %}
4\. Initialize the APT repository. Two options are available: HTTPS and HTTP. HTTPS is recommended because it encrypts package transfers and prevents tampering in transit. Use HTTP only if your environment does not support TLS.
{% hint style="info" %}
**NOTE**
Creating the repository typically takes approximately 20 minutes, depending on hardware performance.
{% endhint %}
**Option 1: HTTPS (recommended)**
If you do not have an existing certificate, generate a self-signed certificate and initialize the repository.
```bash
sudo ./scripts/create_apt_repo.sh init-https repo.local
sudo ./create_apt_repo.sh add-bulk ./deb_packages
```
If you already have a certificate and key, provide the paths directly.
```bash
sudo ./scripts/create_apt_repo.sh init-https repo.example.com /path/to/cert.pem /path/to/key.pem
sudo ./create_apt_repo.sh add-bulk ./deb_packages
```
**Option 2: HTTP (insecure)**
Use this option only if TLS is not available in your environment. Package transfers over HTTP are unencrypted and not verified.
```bash
sudo ./create_apt_repo.sh init
sudo ./create_apt_repo.sh add-bulk ./deb_packages
```
### Step 3: Configure the APT repository on cluster nodes
Update the APT configuration on all management and compute nodes so they can access the private repository.
1. If you are using a self-signed certificate, distribute the CA to each node and update the certificate store.
```bash
sudo cp /path/to/apt_repo.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
```
2. Back up your existing APT sources.
```bash
sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo mkdir -p /etc/apt/sources.list.d.bak
sudo mv /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d.bak/ 2>/dev/null || true
sudo rm /etc/apt/sources.list
```
3. Add the private repository and update the package index.
```bash
echo "deb [trusted=yes] http:/// stable main" | sudo tee /etc/apt/sources.list.d/private-repo.list
sudo apt update
```
### Step 4: Set up a private container registry
The private registry stores all container images required by space.vars.product\_acronym. You set up the registry on a host with internet access and then make it reachable from your air-gapped cluster nodes. To complete this step, you will download the registry setup scripts, initialize the registry, and confirm it is ready to receive images.
1. On a host with internet connectivity, download the registry setup scripts and the image list.
```bash
curl --user-agent "" -O https://pf9-airctl.s3-accelerate.amazonaws.com/latest/pcdv-images.txt
curl --user-agent "" -O https://pf9-airctl.s3.us-west-1.amazonaws.com/latest/sample_scripts/setup_registry.sh
curl --user-agent "" -O https://pf9-airctl.s3.us-west-1.amazonaws.com/latest/push-images.sh
```
2. Make the setup script executable and run it to initialize the registry.
```bash
chmod +x setup_registry.sh
./setup_registry.sh
```
3. When prompted, provide the following inputs to configure the registry.
| Prompt | Example | Description |
| ---------------- | --------------------- | ---------------------------- |
| Registry host IP | `10.10.13.145` | IP of the registry server |
| Registry domain | `registry.pf9.io` | DNS name used by the cluster |
| Username | `` | Registry login username |
| Password | `` | Registry login password |
**Example session:**
```bash
Enter the registry host IP address: 10.10.13.145
Enter the registry domain: registry.pf9.io
Enter the username: admin
Enter the password:
```
### Step 5: Push images to the private registry
In this step, you push all required space.vars.product\_acronym container images to the private registry you created in Step 4. Before Docker can communicate with a registry that uses a self-signed certificate, you must add the registry's CA certificate to Docker's trust store. Without this, Docker rejects the connection and the image push fails.
{% hint style="info" %}
**NOTE**
Docker must be installed on the host machine before you can push images.
{% endhint %}
1. Add the registry's CA certificate to Docker's trust store.
```bash
sudo mkdir -p /etc/docker/certs.d/:443
sudo cp /usr/local/share/ca-certificates/ca.crt /etc/docker/certs.d/:443/ca-cert.pem
```
2. Restart the Docker service so the certificate change takes effect.
```bash
sudo systemctl restart docker
```
3. Push the images to the private registry using the `push-images.sh` script.
```bash
./push-images.sh \
--images-file-path \
--private-registry-url \
--private-registry-username= \
--private-registry-password=
```
### Step 6: Install the OpenSSL dependency
space.vars.product\_acronym requires a specific build of OpenSSL that is not available through the standard Ubuntu package repositories. In this step, you download the pre-built package from the Platform9 artifact store, transfer it to each cluster node, and install it. The checksum verification confirms that the package was not corrupted in transit.
1. On a host with internet access, download the OpenSSL package.
```bash
curl --user-agent "" \
https://pf9-airctl.s3-accelerate.amazonaws.com/openssl-smcp-ubuntu/openssl_3.0.7-1_amd64.deb \
--output openssl_3.0.7-1_amd64.deb
```
2. Transfer the downloaded package to each cluster node.
3. On each node, verify the MD5 checksum to confirm the package integrity.
```bash
md5sum openssl_3.0.7-1_amd64.deb | grep 706caf \
|| { echo "MD5 checksum does not match, exiting."; exit 1; }
```
4. Install the OpenSSL package and configure the library path.
```bash
# Install the OpenSSL package
sudo dpkg -i openssl_3.0.7-1_amd64.deb \
|| { echo "Failed to install OpenSSL, exiting."; exit 1; }
# Add the OpenSSL library path
echo "/usr/local/ssl/lib64" | sudo tee /etc/ld.so.conf.d/openssl-3.0.7.conf
# Refresh the dynamic linker cache
sudo ldconfig -v
# Create a symbolic link to the new OpenSSL binary
sudo ln -sf /usr/local/ssl/bin/openssl /usr/bin/openssl
```
5. Verify that the correct version of OpenSSL is active on the node.
```bash
openssl version | grep 3.0.7 \
|| { echo "OpenSSL version does not match, exiting."; exit 1; }
```
Confirm that the output matches the following before continuing.
```
OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
```
### Step 7: Install required packages
`cgroup-tools` is a Linux utility that space.vars.product\_acronym requires to manage and interact with control groups on each cluster node. Run the following command on each node to update the package index and install `cgroup-tools` in a single operation.
```bash
apt-get update -y && apt-get install cgroup-tools -y
```
### Step 8: Deploy space.vars.product\_acronym
With the private registry and APT repository in place, you can now run the space.vars.product\_acronym installer. Complete the following steps on a management node.
{% hint style="info" %}
**NOTE**
If the registry hostname is not resolvable via DNS in your environment, add the hostname to `/etc/hosts` on each node before proceeding.
{% endhint %}
1. On a system with internet connectivity, download the installer artifacts.
```bash
curl --user-agent "" \
https://pf9-airctl.s3-accelerate.amazonaws.com/latest/index.txt | \
awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/latest/" $NF "\" -o ${HOME}/" $NF}' | bash
```
2. Copy all downloaded artifacts to the management node, then make the installer executable.
```bash
chmod +x ./install-pcd.sh
```
3. Run the installer using the version number from `version.txt`.
```bash
./install-pcd.sh $(cat version.txt)
```
4. Create a symlink so that `airctl` is available globally on the management node.
```bash
sudo ln -s /opt/pf9/airctl/airctl /usr/bin/airctl
```
5. Generate the configuration file for your management cluster. Use single-master for a POC environment or multi-master for production.
{% hint style="info" %}
**NOTE**
Before running this command, copy the `ca.crt` file from the registry to the management node. Provide its path using the `--custom-registry-ca-cert-path` flag.
{% endhint %}
```bash
airctl configure \
-4 \
-f \
-e \
-i \
--master-vip4 \
-v ens3 \
-r \
-k \
-n /mnt/gnocchi \
-p hostpath-provisioner \
--custom-registry-url https://registry.pf9.io \
--custom-registry-username \
--custom-registry-password "" \
--custom-registry-ca-cert-path \
--custom-registry-path-overrides \
--enable-pcd-chart-bundle \
--verbose
```
### Next steps
After completing the air-gapped setup, proceed to [Install](/private-cloud-director/getting-started/self-hosted/self-hosted-install) to deploy the management cluster and install the space.vars.product\_acronym management plane using `airctl`.
# Upgrade Management Plane and Hosts
{% hint style="info" %}
**Upgrading from 2026.1 to 2026.4?** An in-place upgrade is not supported for this version transition. Customers upgrading from **2026.1 to 2026.4** must follow the migration guide:
[Migrating from nodeletd to k3s-Based Clusters](/private-cloud-director/getting-started/self-hosted/migrating-from-nodeletd-to-k3s-based-clusters)
{% endhint %}
The upgrade of the self-hosted space.vars.product\_name is a two-phase, sequential process:
* **Management Plane:** Upgrading the Management Plane updates core services, management APIs, region-specific configurations, and orchestration components.
* **Hosts**: Once the management plane has been upgraded, hosts in each region are being upgraded to ensure compatibility, seamless communication, and completion of the overall upgrade process
Before you begin the upgrade, ensure you meet all prerequisites.
{% hint style="info" %}
**Detailed upgrade runbook and verification checklist**
This page documents the `airctl` commands to upgrade a self-hosted deployment. For deployment-model-agnostic operational guidance — pre-upgrade environment checks, host upgrade ordering, Ubuntu 22.04-to-24.04 OS upgrade caveats, and failure recovery — see the [Host Upgrade Runbook](/private-cloud-director/upgrade/host-upgrade-runbook).
After upgrading, use the [Post-Upgrade Verification](/private-cloud-director/upgrade/post-upgrade-verification) checklist to confirm all services are healthy and to re-enable VM HA and DRR.
{% endhint %}
## Prerequisites for Upgrade
Before you upgrade, complete the following checks to ensure your environment is ready.
### Verify Management Plane Health
Before you upgrade, verify that the Management Plane is healthy. Run `airctl status` and confirm that `region health` shows as `Ready`. If any regions are not ready, resolve all issues before continuing. For example, pods may be in a `CrashLoopBackOff` state.
### Manual Backup
You must perform a [Manual Backup Procedure](/private-cloud-director/getting-started/self-hosted/backup-and-restore#manual-backup-procedure) of your management plane infrastructure before starting the upgrade. This backup is critical for recovery.
## Upgrade Management Plane
Upgrade your space.vars.product\_name environment to access the latest features, security patches, and performance improvements through this two-step process.
### Upgrade Management Plane
Updating the Management Plane running on the Management Cluster ensures multiple components are automatically upgraded as part of an `airctl upgrade` process.
Here are the components that will be affected.
* Core services and applications.
* Management APIs and user interfaces.
* Region-specific configurations and services.
* Service coordination and orchestration components.
#### The Impact of Management Plane Upgrade
{% hint style="info" %}
**NOTE**
Temporary service interruptions may occur during the upgrade process.
{% endhint %}
* The user interface will be unavailable during the upgrade.
* All API services will be temporarily unavailable.
* Create, Read, Update, and Delete operations will not be possible during this time.
* **Running workloads on VMs will not be impacted.**
#### Step 1: Download and run the Installer Artifacts
`airctl` is the command-line installer for space.vars.product\_name. Run the following commands **only on a verified management cluster node** to download `airctl` with the required installation artifacts.
{% hint style="info" %}
**Air-gapped environments only**
If the management node does not have internet access, download the installer artifacts on a jump host with internet connectivity and transfer them to the management node.
Follow the artifact download steps from the air-gapped installation guide, then skip this step and continue using the copied artifacts.
{% endhint %}
1. Download the Installer Script and Artifacts using the following command.
```bash
curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/latest/index.txt | awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/latest/" $NF "\" -o ${HOME}/" $NF}' | bash
```
You can choose to download a specific version of the installer script and artifact.
For example, you can replace `latest` with a specific version, such as `v-2026.1.1-4444155`
Here is an example of a modified command:
{% tabs %}
{% tab title="Example" %}
```bash
curl --user-agent "" https://pf9-airctl.s3-accelerate.amazonaws.com/v-2026.1.1-4444155/index.txt | \
awk '{print "curl -sS --user-agent \"\" \"https://pf9-airctl.s3-accelerate.amazonaws.com/v-2026.1.1-4444155/" $NF "\" -o ${HOME}/" $NF}' | bash
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**NOTE**
Replace `` with your user key.
{% endhint %}
2. Make the Installer Executable using the following command.
```bash
chmod +x ./install-pcd.sh
```
3. Run the Installation Script using the following command.
```bash
./install-pcd.sh `cat version.txt`
```
The command installs the specific version of `airctl` based on the `version.txt` file.
4. Create symbolic links for `airctl` and its configuration file:
```bash
sudo rm -f /usr/bin/airctl # delete any existing file or symlink
sudo ln -s /opt/pf9/airctl/airctl /usr/bin/airctl
ln -s /opt/pf9/airctl/conf/airctl-config.yaml $HOME/airctl-config.yaml
```
#### Proxy Configuration (Optional)
If your environment uses a network proxy, update the values `/opt/pf9/airctl/conf/helm_values/bork.template.yml` again, as the changes would be lost after a new build is installed. Update it again as shown below:
```bash
cat /opt/pf9/airctl/conf/helm_values/bork.template.yml | grep proxy
# Sample output:
https_proxy: "http://squid.platform9.horse:3128"
http_proxy: "http://squid.platform9.horse:3128"
no_proxy: "10.149.106.11,10.149.106.12,10.149.106.13,10.149.106.14,10.149.106.15,10.149.106.16,127.0.0.1,10.20.0.0/22,localhost,::1,.svc,.svc.cluster.local,10.21.0.0/16,10.20.0.0/16,.cluster.local,.platform9.localnet,.default.svc"
```
The `no_proxy` list should include the `master-ips`, `worker-ips`, `external-ip4`, `master-vip4`, and any other addresses where traffic should not be routed through a proxy server.
{% hint style="info" %}
**NOTE**
The upgrade-cluster step is skipped in this release, as the Kubernetes version remains unchanged at **v1.30**.
{% endhint %}
{% hint style="info" %}
**Air-gapped environments only**
Ensure that all required container images for the target version are already available in the private registry accessible by the cluster nodes before running the upgrade.
If not already done, download the image list for the target version and push the images to your private registry. Refer to the air-gapped installation guide for detailed steps.
{% endhint %}
#### Step 2: Upgrade all regions
To upgrade **all** regions set up on your space.vars.product\_name, execute the following command.
```bash
airctl upgrade
```
**Optionally**, you can upgrade a specific region by replacing `` with your target region.
```bash
airctl upgrade --region
```
Here is the modified example command.
{% tabs %}
{% tab title="Example" %}
```json
# airctl upgrade --region Region1
INFO rollback state directory /tmp/airctl_ddu_backup_Region1_240020739
INFO Saving the helm revisions to the state file
INFO --- backing up region--- Region1
INFO Archive created successfully for Region1. Backup backup.tar.gz saved to /tmp/airctl_ddu_backup_Region1_240020739
INFO --- moving old state file to /tmp/airctl_ddu_backup_Region1_240020739/state.yaml ---
INFO --- moving old bork_values.yaml file to /tmp/airctl_ddu_backup_Region1_240020739/bork_values.yaml ---
SUCCESS Upgrading region Region1
upgrade done
```
{% endtab %}
{% endtabs %}
To monitor and diagnose the upgrade logs, add `--verbose`.
Here is the sample command.
```bash
airctl upgrade --verbose
```
#### Verify: Upgrade Success on Management Plane
After completing the upgrade for the management plane, verify that you can continue to access the user interface, and then verify the deployment status across all regions.
Run the following command:
```bash
airctl status
```
Here is the sample output.
{% tabs %}
{% tab title="Example" %}
```bash
------------- deployment details ---------------
fqdn: airctl-1-4206457-802.platform9.localnet
region: airctl-1-4206457-802
deployment status: ready
region health: ✅ Ready
version: PCD v-2026.1.1-4444155
-------- region service status ----------
desired services: 30
ready services: 30
------------- deployment details ---------------
fqdn: airctl-1-4206457-802-mel.platform9.localnet
region: airctl-1-4206457-802-mel
deployment status: ready
region health: ✅ Ready
version: PCD v-2026.1.1-4444155
-------- region service status ----------
desired services: 84
ready services: 84
```
{% endtab %}
{% endtabs %}
After a successful upgrade and verification, space.vars.product\_name it **does not support rolling back** to a previous version.
## Upgrade the Hosts
After the management plane is successfully upgraded, the hosts running in each of your regions must be updated. The agents running on the Host manage communication between hosts and the management plane.
### The Impact from Hosts Upgrade
* No `servHostavailability` is expected during the host upgrade.
* Workloads running on VMs will continue to operate without disruption.
#### Step 1: Record the Current Version of Packages on the Host
Before you begin the upgrade, execute the following command to record the current package versions:
{% tabs %}
{% tab title="Example" %}
```bash
$ airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml
Getting host statuses...
Getting host statuses for region: [REGION_NAME]
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| Host Name | IP Addresses | Host ID | Host Agent Version | Status | Agent Status | Apps |
| [HOSTNAME1] | 172.29.32.34, 192.168.122.1, 10.0.11.10 | [HOST1_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-cindervolume-config:2026.1.1-4444155|
| | | | | | | pf9-cindervolume-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-glance-role:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| [HOSTNAME2] | 172.29.32.110, 192.168.122.1, 10.0.11.102 | [HOST2_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SUCCESS getting host statuses...
```
{% endtab %}
{% endtabs %}
#### Step 2: Upgrade Hosts
To upgrade the hosts in a specific region, execute the following commands on the management cluster node:
{% hint style="info" %}
**NOTE**
The `upgrade-hosts` command times out after **1800 seconds (30 minutes)** by default. You can override this default by adding the `hostupgradetimeout` parameter in the `airctl-config.yaml` file. To prevent timeouts when upgrading a large number of hosts, increase this value proportionally — for every **50 hosts**, add **30 minutes (1800 seconds)** to the `hostupgradetimeout` value.
{% endhint %}
```bash
airctl upgrade-hosts --region
```
The command triggers the creation of a `host-upgrade-xxxx` pod in the corresponding region namespace. You can monitor the upgrade progress or verify its success by checking the pod status and logs:
```bash
# List the host upgrade pods
kubectl get pods -n | grep host-upgrade
# View logs of the host upgrade pod
kubectl logs -n
```
Here is a sample output:
{% tabs %}
{% tab title="Example" %}
```bash
host-upgrade-1747919688-g5rk5 5/5 Completed 0 83m
kubectl logs -n example-onprem-region1 host-upgrade-1747919688-g5rk5
```
{% endtab %}
{% endtabs %}
If a host fails to upgrade, run the following command to rerun the upgrade that the user executes using **IP.**
```bash
airctl upgrade-hosts --region --host-ips ""
```
#### Step 3: Verify Upgrade Status of the Hosts
After the upgrade, confirm that the host packages have been successfully updated by running the following command.
{% tabs %}
{% tab title="Example" %}
```bash
$ airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml
Getting host statuses...
Getting host statuses for region: [REGION_NAME]
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| Host Name | IP Addresses | Host ID | Host Agent Version | Status | Agent Status | Apps |
| [HOSTNAME1] | 172.29.32.34, 192.168.122.1, 10.0.11.10 | [HOST1_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-cindervolume-config:2026.1.1-4444155 |
| | | | | | | pf9-cindervolume-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-glance-role:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| [HOSTNAME2] | 172.29.32.110, 192.168.122.1, 10.0.11.102 | [HOST2_UUID] | v-2026.1.1-4444155.6f598c0 | ok | running | pf9-ostackhost:2026.1.1-4444155 |
| | | | | | | pf9-neutron-base:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-controller:2026.1.1-4444155 |
| | | | | | | pf9-neutron-ovn-metadata-agent:2026.1.1-4444155 |
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SUCCESS getting host statuses...
```
{% endtab %}
{% endtabs %}
Compare the output with the previous recording to ensure the packages were updated as expected.
## Recovery: Upgrade Failure
#### Step 1. Stop the Failed Management Cluster
```bash
airctl stop --verbose
```
This command shuts down all components of the existing management cluster. `--verbose` helps verify the detailed output of all stopped services.
#### Step 2. Delete the Management Cluster Configuration
```bash
airctl unconfigure-du --force --verbose
```
This command removes all existing configuration files and metadata associated with the management cluster. Using `--force` enables overriding any locked or incomplete states.
#### Step 3: Delete the Existing Management Cluster
```bash
airctl delete-cluster --verbose
```
This command permanently deletes the management cluster resources. Ensure that you have backed up all critical data before running this command
#### Step 4. Create a New Management Cluster
```bash
airctl --config /opt/pf9/airctl/conf/airctl-config.yaml create-cluster --verbose
```
The command creates a new management cluster with the same configuration as before
#### Step 5. Restore from the Backup
```bash
airctl restore --backupdir --verbose
```
Replace ``with the actual path of your stored backup. The command restores the environment from the backup.
{% hint style="warning" %}
**Warning**
After a management plane upgrade, a rollback to a pre-upgrade version cannot be performed if any hosts have been upgraded as well.
{% endhint %}
# Migrating from nodeletd to k3s-Based Clusters
### Overview
This guide describes the recommended procedure for migrating an existing 2026.1 **nodeletd-based** Self-hosted Private Cloud Director deployment to a **k3s-based** cluster ( release 2026.4). The migration uses a backup-and-restore approach and is the supported upgrade path for on-premises customers in this release.
The high-level flow is:
1. Install the new release artifacts
2. Back up the existing (2026.1) deployment
3. Unconfigure and delete the existing cluster
4. Reconfigure and create a new k3s cluster
5. Restore the backup
> **Note:** This process involves deliberate cluster destruction and recreation. Plan for a maintenance window before proceeding.
***
### Prerequisites
Before starting the migration, confirm the following:
* You have access to the airctl management host.
* You have reviewed and noted all values from your current `/opt/pf9/airctl/conf/airctl-config.yaml` file and your existing nodelet bootstrap configuration ( `/opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml` ) . You will need these values when reconfiguring the deployment.
* You have confirmed that a valid **restore target** exists (an accessible backup location or S3-compatible store).
* The **new release artifacts and packages** have been installed on the management host exactly as you would during a standard upgrade. Refer to the Upgrade Guide for package installation steps.
* You have read and understood the Important Considerations section below.
***
### Important Considerations
> ⚠️ **This migration involves irreversible steps.** Specifically, `airctl unconfigure-du` and `airctl delete-cluster` are **destructive and cannot be undone**. Ensure your backup is complete and verified before proceeding past Step 3.
* **Management Plane Downtime is expected.** This migration requires the cluster to be fully destroyed before the new k3s cluster is created. Plan for a maintenance window sized to your environment's restore time. In most environments, expect **several hours of downtime**.
* **There is no in-place rollback.** Once the cluster is deleted, rollback requires restoring from the backup taken in this procedure. Ensure the backup completes successfully and is accessible from the restore target before continuing.
* **Configuration values must be preserved.** During reconfiguration, you will re-run `airctl configure` using the same or equivalent inputs as the original installation. Derive these values from:
* Your current `airctl-config` file
* Your existing nodelet bootstrap configuration
* **Preserve your `options.json` file.** The `options.json` file, located at `/opt/pf9/airctl/conf/options.json`, controls deployment-specific settings such as storage class configuration, audit settings, and other platform options. This file is generally carried over by a new installation, but it is good practice to verify its contents before triggering the restore. If it is missing or outdated, recreate or update it to reflect your intended configuration before running the restore command.
* **Verify the backup before proceeding.** A corrupted or incomplete backup will prevent successful restore. Confirm the backup artifact is intact and readable before deleting the existing cluster.
***
### Migration Procedure
#### Step 0 — Back Up Airctl Host Configuration Files
Before installing new release artifacts, back up `options.json` and any other customized configuration files from the airctl host. The install script may overwrite `options.json` — preserving a copy now ensures you can restore your site-specific settings in Step 5a.
```bash
mkdir -p /tmp/airctl-conf-backup
# Back up the main options file
cp /opt/pf9/airctl/conf/options.json /tmp/airctl-conf-backup/options.json
# If you have other customized configuration files in /opt/pf9/airctl/conf/,
# copy them here as well before proceeding.
```
Keep this backup accessible on the host. You will reference it in Step 5a (Restore options.json Customizations).
***
#### Step 1 — Install New Release Artifacts
Install the new packages as you would for a standard upgrade. Do not restart or reconfigure the deployment at this stage.
Refer to the [Upgrade Guide](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/upgrade#step-1-download-and-run-the-installer-artifacts) for the artifact installation step — specifically, the step that runs the install script. **Follow only the artifact installation step.** Do not run `airctl upgrade`, `airctl start`, or any subsequent steps from the Upgrade Guide. Return to this guide after the install script completes.
> ⚠️ Ensure you have completed Step 0 before running the install script.
***
#### Step 2 — Back Up the Existing Deployment
Take a full backup of the existing deployment before making any configuration changes.
```bash
/opt/pf9/airctl/airctl backup --outdir /tmp/backup-mgmt/ --verbose
```
> ⚠️ **Do not proceed until the backup completes successfully.** Verify that the backup artifact exists at the expected destination and that it is not empty or corrupt. Keep note of the backup path or identifier — you will need it in Step 7.
***
#### Step 3 — Unconfigure the Existing Deployment
Before proceeding, verify that the deployment is healthy. This is the last safe stopping point before an irreversible operation.
```bash
airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
> ⚠️ Do not proceed if any region is degraded. Resolve issues first. The next command cannot be undone without a valid backup.
Uninstall the current deployment. This step stops services and clears the active configuration state.
```bash
/opt/pf9/airctl/airctl unconfigure-du
```
***
#### Step 4 — Delete the Existing Cluster
Delete the existing nodeletd cluster. This step is **irreversible**.
```bash
/opt/pf9/airctl/airctl delete-cluster
```
***
#### Step 5 — Reconfigure the Deployment
Re-run `airctl configure` using the same inputs that were used during the original installation.
For a full description of the `airctl configure` parameters and prompts, refer to the [Installation Guide](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/self-hosted-install#configure-airctl). Use the values from your saved `airctl-config` and nodelet bootstrap configuration as reference. Have both files open in a separate terminal session before proceeding.
> **Note:** The configuration inputs required here are identical to those used during initial installation. Do not use default or example values — use the values specific to your environment.
**Proxy configuration:** If your environment requires a proxy, configure it after `airctl configure` completes. Refer to the [Proxy Configuration](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/self-hosted-install#proxy-configuration-optional) section of the Installation Guide for instructions.
**Custom certificates:** If your original deployment used custom SSL/TLS certificates, you must supply them again during reconfiguration. Pass the certificate and key paths as arguments to `airctl configure`:
```bash
/opt/pf9/airctl/airctl configure \
--user-cert-path /path/to/service.crt \
--user-key-path /path/to/service.key
```
Alternatively, export them as environment variables before running the command:
```bash
export USER_CERT_PATH="/path/to/service.crt"
export USER_KEY_PATH="/path/to/service.key"
```
After `airctl configure` completes, verify that `user_cert_path` and `user_key_path` are present in `/opt/pf9/airctl/conf/airctl-config.yaml`. For full details, see [Using Custom Certificates](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/using-custom-certificates).
***
#### Step 5a — Restore options.json Customizations
After `airctl configure` completes, `options.json` is regenerated with default values. Verify the new file against your backup and merge back any site-specific customizations.
```bash
# Review differences between your backup and the newly generated file
diff /tmp/airctl-conf-backup/options.json \
/opt/pf9/airctl/conf/options.json
```
For each value that differs and should reflect your site configuration, update the new file. You can edit it directly or use `jq` to restore specific keys. For example:
```bash
jq --slurpfile bk /tmp/airctl-conf-backup/options.json \
'.rabbitmq_clustering_enabled = $bk[0].rabbitmq_clustering_enabled' \
/opt/pf9/airctl/conf/options.json > /tmp/options.merged.json \
&& mv /tmp/options.merged.json /opt/pf9/airctl/conf/options.json
```
If you backed up other customized configuration files in Step 0, restore those now before continuing.
***
#### Step 5b — Verify and Recreate the airctl Symlink
After reconfiguration, confirm that the `airctl` binary symlink at `/usr/local/bin/airctl` points to the newly installed version.
```bash
# Verify the symlink
ls -la /usr/local/bin/airctl
# Recreate if missing or pointing to an old path
sudo ln -sf /opt/pf9/airctl/airctl /usr/local/bin/airctl
# Confirm the version is correct
airctl version
```
If your site uses a convenience symlink to the configuration file, recreate it now:
```bash
sudo ln -sf /opt/pf9/airctl/conf/airctl-config.yaml /etc/airctl-config.yaml
```
***
#### Step 6 — Validate Post-Configure State
After `airctl configure` completes, verify that the environment is correctly initialized before creating the cluster.
**Verify the updated** `/opt/pf9/airctl/conf/airctl-config.yaml` file.
Confirm that the configuration reflects the values you provided and that there are no unexpected changes.
**Verify the k3s bootstrap configuration:**
Confirm that the k3s bootstrap configuration has been created at the expected path: `/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml`
***
#### Step 7 — Create the New k3s Cluster
Create the new k3s-based cluster.
/opt/pf9/airctl/airctl create-cluster --verbose
Wait for the cluster creation to complete. Monitor the output for errors. Do not interrupt this process.
***
#### Step 8 — Restore the Backup
Once the new cluster is running, restore your data from the backup taken in Step 2.
{% hint style="info" %}
**RabbitMQ clustering:** The restore process enables RabbitMQ clustering by default. If you wish to disable it, update `/opt/pf9/airctl/conf/options.json` before proceeding with the restore by setting:
```json
"rabbitmq_clustering_enabled": "false"
```
{% endhint %}
Refer to the [Backup and Restore Guide](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/backup-and-restore) for detailed restore instructions.
**Quick reference — restore command:**
```bash
/opt/pf9/airctl/airctl restore --backupdir --verbose
```
Replace `` with the path or identifier recorded in Step 2.
> **Note:** The restore process may take significant time depending on the size of your deployment. Do not interrupt it once started.
***
### Validation Steps
After the restore completes, verify that the migration was successful:
1. **Confirm deployment health** — The deployment may take up to 10–20 minutes to fully initialize after restore. Once the initialization period has passed, confirm all regions are healthy:
```bash
airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
If any region remains degraded after 20 minutes, collect diagnostics and contact Platform9 Support.
2. **Confirm cluster node health** — Check that all cluster nodes report as healthy:
```bash
kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes
```
3. **Confirm management plane availability** — Log in to the Private Cloud Director UI and confirm that the dashboard loads and all services are visible.
4. **Verify workloads and data** — Confirm that previously managed hosts, clusters, and configurations are present and accurate.
***
### **Upgrade hosts**
Once the restore is complete, the management plane upgrade is considered done. You must then upgrade the hosts in each region to ensure compatibility and complete the overall upgrade process. Refer to the [Upgrade Management Plane and Hosts](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/upgrade#upgrade-the-hosts) guide for host upgrade instructions.
***
### Post-Migration Notes
* The migration replaces the underlying cluster runtime from **nodeletd** to **k3s**. Application-level behavior and the management plane interface remain unchanged for end users.
* **kubeconfig path change:** The management cluster kubeconfig has moved. Update any scripts, `.bashrc` aliases, or CI pipelines that reference the previous path.
| Cluster type | kubeconfig path |
| --------------------------- | ------------------------------------------------- |
| nodeletd (before migration) | `/etc/nodelet/airctl-mgmt/certs/admin.kubeconfig` |
| k3s (after migration) | `/etc/rancher/k3s/k3s.yaml` |
```bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes
```
* **kubectl binary path change:** The kubectl binary is now located at `/opt/pf9/airctl/bin/kubectl`. Update any scripts or aliases that invoke kubectl using an explicit path.
```bash
/opt/pf9/airctl/bin/kubectl version --client
```
* If you encounter issues during or after restore, contact Platform9 Support and provide the `airctl-config` in use, and any relevant log output.
* Review the [Release Notes](https://docs.platform9.com/release-notes/april-2026-release) for known issues or post-migration configuration recommendations specific to this release.
***
### Rollback Considerations
There is no automated rollback for this migration. If the migration fails after the cluster has been deleted (Step 4 or later), recovery requires:
1. Re-running `airctl unconfigure-du --` with `--force` flag
2. Restoring from the backup created in Step 2.
If the backup cannot be restored, contact Platform9 Support immediately. **Do not delete or overwrite the backup artifact under any circumstances.**
# Cluster Scaling & Other Operations
This document describes steps to scale your K3s management cluster that is part of your self-hosted space.vars.product\_name deployment.
## Scale Up Management Cluster
Let's consider that we have 3 master nodes as part of your management cluster, with IP addresses ``1.1.1.1, 2.2.2.2 `and` 3.3.3.3``.
{% tabs %}
{% tab title="K3s Bootstrap Config File" %}
```bash
$ cat /opt/pf9/airctl/conf/k3s-bootstrap-config.yaml
...
masterNodes:
- nodeName: 1.1.1.1
- nodeName: 2.2.2.2
- nodeName: 3.3.3.3
```
{% endtab %}
{% endtabs %}
Lets say that you want to scale up the management cluster to 5 nodes, and that `4.4.4.4 & 5.5.5.5` are the IP addresses of the 2 new nodes to be added.
To scale up the number of cluster nodes to 5:
* Configure the [pre-requisites](/private-cloud-director/getting-started/self-hosted/self-hosted-pre-requisites) on the two new nodes.
* Edit the cluster bootstrap configuration file `/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml` and add the two new IP addresses to the section called `masterNodes` in the file.
* Finally, run the `airctl` command shown below to scale up the cluster.
{% hint style="info" %}
**Note**
`airctl` expects the node count to be always odd in the cluster bootstrap configuration file `/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml`
{% endhint %}
{% tabs %}
{% tab title="k3s Bootstrap Config File" %}
```bash
$ cat /opt/pf9/airctl/conf/k3s-bootstrap-config.yaml
...
masterNodes:
- nodeName: 1.1.1.1
- nodeName: 2.2.2.2
- nodeName: 3.3.3.3
- nodeName: 4.4.4.4
- nodeName: 5.5.5.5
```
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="Scale Command" %}
```bash
airctl scale-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```
{% endtab %}
{% endtabs %}
Now verify that the management cluster has scaled up by querying the cluster nodes.
{% tabs %}
{% tab title="Cluster State Post Scale Up" %}
```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
1.1.1.1 Ready control-plane,etcd,master 4d23h v1.33.9+k3s1
2.2.2.2 Ready control-plane,etcd,master 4d23h v1.33.9+k3s1
3.3.3.3 Ready control-plane,etcd,master 4d23h v1.33.9+k3s1
4.4.4.4 Ready control-plane,etcd,master 5m42s v1.33.9+k3s1
5.5.5.5 Ready control-plane,etcd,master 5m40s v1.33.9+k3s1
```
{% endtab %}
{% endtabs %}
## Management Cluster Status
To check the status of your management cluster, run the following command:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml --region
```
{% endtab %}
{% endtabs %}
Sample output:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml --region foo-region1
# Sample output:
------------- deployment details ---------------
fqdn: foo-region1.bar.io
cluster: foo-bork.bar.io
region: foo-region1
task state: ready
version: v-5.12.0-3479469
-------- region service status ----------
desired services: 45
ready services: 45
```
{% endtab %}
{% endtabs %}
## Scale Down Management Cluster
Now let's assume that we want to remove nodes `2.2.2.2` and `3.3.3.3` from the management cluster. To scale down the cluster:
* Edit the cluster bootstrap configuration file and remove the IP addresses of the two nodes.
* Then run the `airctl` command as shown below to scale down the cluster.
{% tabs %}
{% tab title="K3s Bootstrap Config File" %}
```bash
$ cat /opt/pf9/airctl/conf/k3s-bootstrap-config.yaml
...
masterNodes:
- nodeName: 1.1.1.1
- nodeName: 4.4.4.4
- nodeName: 5.5.5.5
```
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="Scale Command" %}
```bash
airctl scale-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```
{% endtab %}
{% endtabs %}
* Cluster state post scale down operation completion:
{% tabs %}
{% tab title="Cluster State Post Scale Down" %}
```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
1.1.1.1 Ready control-plane,etcd,master 4d23h v1.33.9+k3s1
4.4.4.4 Ready control-plane,etcd,master 5m42s v1.33.9+k3s1
5.5.5.5 Ready control-plane,etcd,master 5m40s v1.33.9+k3s1
```
{% endtab %}
{% endtabs %}
## Uninstall Self-Hosted Deployment
To uninstall the specific region of your self-hosted deployment, run the following command. If you want to uninstall all regions, just remove `--region` flag.
{% tabs %}
{% tab title="Bash" %}
```bash
airctl unconfigure-du --config /opt/pf9/airctl/conf/airctl-config.yaml --region --force
```
{% endtab %}
{% endtabs %}
This command will uninstall and remove configured regions along with all infrastructure software like consul, vault, percona, k8sniff, etc.
If you plan to reuse the same nodes to deploy a new self-hosed space.vars.product\_name environment, make sure to also run the following command on all nodes first.
{% tabs %}
{% tab title="Bash" %}
```bash
rm -rf airctl* install-pcd.sh nodelet* options.json pcd-chart.tgz /opt/pf9/airctl/ .airctl/
```
{% endtab %}
{% endtabs %}
# Backup and Restore Management Plane
This guide provides steps for backing up and restoring the self-hosted space.vars.product\_name management plane in disaster recovery scenarios. The procedures include both manual and automated backup methods, as well as manual restoration process.
{% hint style="info" %}
**NOTE**
When restoring the management plane, ensure it's done on a K3s management cluster that is separate from the cluster where the backup was generated.
{% endhint %}
## Prerequisites
#### System Requirements
* Access to the K3s management cluster
* Installed and configured `airctl` binary
* Valid `airctl` configuration file at `/opt/pf9/airctl/conf/airctl-config.yaml`
* Root or sudo access to the management node
#### For S3 Backup Storage
* AWS credentials with S3 bucket access
* Existing S3 bucket for backup storage
* AWS CLI configured (for verification purposes)
## Important Considerations
1. The restoration process must be performed on a separate K3s management cluster that is different from the management cluster where the backup was generated.
### Manual Backup Procedure
1. Create a backup directory:
{% tabs %}
{% tab title="Bash" %}
```bash
mkdir -p /tmp/backup-mgmt/
```
{% endtab %}
{% endtabs %}
2. Execute the airctl backup command:
{% hint style="info" %}
**Info**
Execute the following commands as a non-root user.
{% endhint %}
{% tabs %}
{% tab title="Bash" %}
```bash
airctl backup --outdir /tmp/backup-mgmt/ --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
Use --region \ parameter if you intend to back up only a specific region. If not specified, all the regions will be included in the backup.
{% endhint %}
3. Verify backup contents:
{% tabs %}
{% tab title="Bash" %}
```bash
tar tvf /tmp/backup-mgmt/backup.tar.gz
```
{% endtab %}
{% endtabs %}
The backup archive should contain:
* `state_backup.yaml`: System state configuration
* `bork_values_backup.yaml`: Kubernetes management cluster configuration
* `consul.snap`: Consul snapshot
* `mysql_dump_Infra.sql`: Infrastructure database backup
* `mysql_dump_Region1.sql`: Region-specific database backup
* `ovn-north-backup` & `ovn-south-backup` : Ovn database backup
## Automated Backup Configuration
The space.vars.product\_name management plane includes an automated backup system that protects your data and configuration. This system creates regular backups and can store them both locally and in Amazon S3.
Learn how to verify backup operations and configure S3 storage for your backups.
### Understanding the backup system
When you install space.vars.product\_name management plane, the system automatically sets up backup protection for you. During installation, it creates a cronjob called `mgmt-plane-backup` in the `pf9-utils` namespace that runs every hour to back up your system.
Your backups get stored in a dedicated storage area called `mgmt-plane-backup-pvc` on your K3s cluster. This storage persists even if pods restart, keeping your backup data safe and accessible.
#### Step 1: Verify backup operations
You can check the status of your backup system at any time using kubectl commands.
1. Run the following command to view the backup cronjob status.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl get cronjob mgmt-plane-backup -n pf9-utils
```
{% endtab %}
{% endtabs %}
This displays when you ran the last backup, confirming your system is working properly.
#### Step 2: Check backup logs
To troubleshoot backup operations or verify successful completion, you can view the backup logs.
1. List the backup pods to find the most recent operation.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl get po -n pf9-utils | grep mgmt-plane-backup
```
{% endtab %}
{% endtabs %}
2. From the output, copy the pod name with the most recent timestamp.
3. View the logs for that specific pod.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl logs mgmt-plane-backup-29167800-l294f -n pf9-utils
```
{% endtab %}
{% endtabs %}
Replace `mgmt-plane-backup-29167800-l294f`with your actual pod name from step 1.
The logs show detailed information about the backup operation, including any errors or success messages.
#### Step 3: Configure S3 backup storage
To enable storing backups in an S3 bucket, you need to configure S3 credentials in a secret named `aws-credentials` in the`pf9-utils` namespace.
Before you begin, consider the following points.
* The backup system stores data in the PVC named `mgmt-plane-backup-pvc` on the K3s cluster and will also upload to your configured S3 location.
* The `mgmt-plane-backup`cronjob runs hourly to ensure regular system backups to both local storage and S3.
1. Run the following command to open and edit `aws-credentials` on the editor.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl edit secret aws-credentials -n pf9-utils
```
{% endtab %}
{% endtabs %}
2. On the `aws-credentials` edit section `data:`
{% tabs %}
{% tab title="Bash" %}
```bash
# Make the edits to the yaml file
data:
AWS_ACCESS_KEY_ID: ""
AWS_SECRET_ACCESS_KEY: ""
AWS_REGION: ""
AWS_S3_PATH: "s3://"
```
{% endtab %}
{% endtabs %}
3. Replace the placeholder values with your actual AWS credentials.
| Placeholder | Replace with |
| ------------------------ | ---------------------------------------- |
| `YOUR_ACCESS_KEY` | Your AWS access key ID |
| `YOUR_SECRET_KEY` | Your AWS secret access key |
| `YOUR_AWS_REGION` | Your AWS region (for example, us-west-2) |
| `YOUR_BUCKET_NAME/PATH/` | Your S3 bucket name and optional path |
3. Save and close the editor.
Once configured, your backups will be stored both locally in the PVC and in your specified S3 bucket location. This provides enhanced data protection and allows for disaster recovery scenarios.
You have successfully configured automated backups for your space.vars.product\_name management plane. Your system now creates regular backups and stores them securely both locally and in Amazon S3.
## Manual Restore Procedure
When you need to restore your space.vars.product\_name management plane, you can access backups stored locally in the PVC or from Amazon S3. This section walks you through both restoration methods.
### Standard Restore (from local PVC)
To restore from local backups, you need to access the backup files stored in the mgmt-plane-backup-pvc. This process involves finding the PVC, locating the underlying storage, and mounting it to access the backup files.
#### Step 1: Locate the backup PVC
Run the following command to find the backup PVC in the `pf9-utils` namespace.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl get pvc -n pf9-utils
```
{% endtab %}
{% endtabs %}
The output displays your backup PVC details. Here is a sample example.
{% tabs %}
{% tab title="Example" %}
```bash
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
mgmt-plane-backup-pvc Bound pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632 100Gi RWO pcd-sc 34h
```
{% endtab %}
{% endtabs %}
#### Step 2: Find the underlying persistent volume
Get the volume name for the backup PVC, by running the following command.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl get pvc mgmt-plane-backup-pvc -n pf9-utils -o jsonpath='{.spec.volumeName}'
```
{% endtab %}
{% endtabs %}
The output returns the persistent volume name. Here is an example.
{% tabs %}
{% tab title="Bash" %}
```bash
pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
```
{% endtab %}
{% endtabs %}
#### Step 3: Get the persistent volume configuration
Describe the persistent volume to find the NFS share information by running the following command.
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl describe pv pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
```
{% endtab %}
{% endtabs %}
In the output, look for the `Source:` section with `VolumeHandle` field. This contains the NFS server and path information that you need for mounting.
#### Step 4: Install NFS utilities
Install the required NFS packages on your system by running the following command.
{% tabs %}
{% tab title="Bash" %}
```bash
ubuntu@sample-vm:~$ sudo apt update && sudo apt -y install nfs-common
```
{% endtab %}
{% endtabs %}
#### Step 5: Mount the NFS share
Create a local directory and mount the NFS share:
{% tabs %}
{% tab title="Bash" %}
```bash
ubuntu@sample-vm:~$ mkdir -p /home/ubuntu/backups
ubuntu@sample-vm:~$ sudo mount -t nfs 10.149.106.253:/mnt/gnocchi /home/ubuntu/backups
```
{% endtab %}
{% endtabs %}
Replace `10.149.106.253:/mnt/gnocchi` with the server and share path from your `VolumeHandle` field.
#### Step 6: Access the backup files
List the available backup files by running the following command
{% tabs %}
{% tab title="Bash" %}
```bash
ubuntu@sample-vm:~$ ls backups/pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
```
{% endtab %}
{% endtabs %}
The backup files will display with timestamps. Here is a sample output.
{% tabs %}
{% tab title="Bash" %}
```bash
backup_20250608_210055.tar.gz backup_20250608_230057.tar.gz backup_20250609_010057.tar.gz
backup_20250609_030054.tar.gz backup_20250609_050055.tar.gz backup_20250608_220057.tar.gz
```
{% endtab %}
{% endtabs %}
Choose the backup file you want to restore based on the timestamp that matches your desired restore point.
#### Step 7: Execute the restore command
Extract and specify the backup directory path, then run the restore command:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl restore --backup-dir --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
Replace ``with the path to your extracted backup directory.
{% hint style="info" %}
**NOTE**
Execute the restore command as a non-root user, similar to the backup procedure.
{% endhint %}
### Restore from S3 Backup
Create and configure the `/etc/default/airctl-backup` file with required AWS parameters, making sure that `AWS_S3_PATH` points specifically to the backup file you want to restore, not just the S3 bucket:
{% tabs %}
{% tab title="Bash" %}
```bash
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
AWS_S3_PATH=s3://
```
{% endtab %}
{% endtabs %}
Execute the S3 restore command:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl restore --s3backup --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**NOTE**
For complete disaster recovery, manually restore Gnocchi metrics data from the original `pcd-sc` persistent volume
{% endhint %}
Verification Steps
Check backup file integrity using MD5 checksum::
{% tabs %}
{% tab title="Bash" %}
```bash
# Generate MD5 checksum for the backup file
md5sum .tar.gz
# Optional: Compare with a pre-recorded checksum
# You can save the MD5 checksum when initially creating the backup
md5sum /root/backup-mgmt/backup.tar.gz > backup-checksum.txt
# Later, verify the backup file matches the original checksum
md5sum -c backup-checksum.txt
```
{% endtab %}
{% endtabs %}
Verify S3 uploads (if configured):
{% tabs %}
{% tab title="Bash" %}
```bash
aws s3 ls s3:///
```
{% endtab %}
{% endtabs %}
Monitor restore progress:
{% tabs %}
{% tab title="Bash" %}
```bash
kubectl logs -f
```
{% endtab %}
{% endtabs %}
## Common Issues
* If AWS credentials are not properly configured, automated S3 backups will continue locally but skip S3 upload.
* Restore operations may take significant time depending on data volume.
* Services may take additional time to start after restore completion.
# Airctl Reference
Airctl is an on-premises Kubernetes orchestrator by Platform9. Comprising a useful command-line utility and related functions that manage the lifecycle of the Management Server on the Management Plane of space.vars.product\_name.
This powerful tool serves as your control center for managing space.vars.product\_name on-premises environment. Airctl provides comprehensive management capabilities that ensure your Kubernetes environment runs smoothly while automatically handling critical infrastructure components.
### What Airctl Does?
* **Cluster Lifecycle Management**: Create, configure, scale, and upgrade K3s management clusters.
* **Backup & Recovery**: Protect your cluster configuration and MySQL database.
### Prerequisites
Before using Airctl, ensure your environment meets these requirements:
* **Operating System**: Ubuntu 22.04 with `sudo` access.
* **Storage**: NFS server (for shared storage scenarios).
* **Resources**: Minimum hardware requirements for your target deployment type.
### Airctl Core Functions
Airctl offers functions that simplify on-premises management.
#### State management
Initializes and configures the Management Plane, creating necessary components automatically when needed.
#### Database Backup
Facilitates MySQL database backup for the Platform9 data store. Optimized for typical three-node cluster configurations.
#### Configuration Management
Manages cluster state through configuration files containing node IP addresses and credentials. These files describe the desired state of your cluster and serve as the foundation for deployment and management operations.
#### CLI Utilities
Provides comprehensive command-line utilities for backup and recovery of cluster state. These utilities ensure you can protect, restore, and maintain your entire management cluster configuration, enabling disaster recovery and environment replication across different deployments.
## Command Reference
### Cluster Management Commands
| **Command** | **Description** | **Use Case** |
| ----------------- | ----------------------------------------------- | -------------------------- |
| `create-cluster` | Create a k3s-based management cluster | Initial cluster deployment |
| `delete-cluster` | Delete a headless management cluster | Cluster decommissioning |
| `scale-cluster` | Scale up or down a k3s-based management cluster | Capacity adjustment |
| `upgrade-cluster` | Upgrade a k3s-based management cluster | Version updates |
| `start` | Start the Management Plane and auto-configure | Normal operations |
| `stop` | Stop the Management Plane | Maintenance windows |
| `status` | Report Management Plane status | Health monitoring |
| `upgrade` | Upgrade the management plane | Normal operations |
| `add-region` | Add a new region to the PCD deployment | Advanced multi tenancy |
### Configuration Commands
| **Command** | **Description** | **Example** |
| ----------------------- | --------------------------------------- | ----------- |
| `configure` | Generate management plane configuration | |
| `update-admin-password` | Set airctl admin password | |
| `get-creds` | Display admin credentials | |
### Backup and Recovery Commands
These CLI utilities provide comprehensive backup and recovery capabilities for both MySQL database and cluster state management.
| **Command** | **Description** | **Output** |
| -------------------- | ------------------------------------------------ | ------------------------- |
| `backup` | Back up management plane state and configuration | Creates tar.gz file |
| `restore` | Restore from backup (includes cluster state) | Restores from tar.gz file |
| `gen-support-bundle` | Generate support bundle | Diagnostic package |
### Maintenance Commands
| **Command** | **Description** | **When to Use** |
| ----------------- | --------------------------------------------------------------------- | -------------------------------- |
| `check` | Verify k3s pre-requisites on all management cluster nodes (see below) | Before installation |
| `upgrade` | Upgrade management plane apps | Version updates |
| `reset-taskstate` | Reset deployment state after resolving errors | Management plane upgrade failure |
#### What `airctl check` Verifies
`airctl --config check` runs a pre-requisites pass on every node listed in the airctl config file. Per node, it confirms:
* The node is **Ready** and reachable.
* **Disk Space** — at least 60 GB free.
* **Swap** is disabled.
* **IPv6 Support** is enabled.
* **Kernel and VM Panic Settings** are configured.
* **Port Connectivity (K3s)** — the ports k3s requires are not already in use: `2379`, `2380`, `6443`, `10250`.
* The **Firewalld** service is in the expected state.
* **Default Route Weights** are correct.
* An **NTP Service** is installed and the node clock is synchronized (see [Pre-requisites](/private-cloud-director/getting-started/pre-requisites)).
The command reports `✓` (pass) or `x` (fail) per check on each node, and a per-node summary at the end. Address any failures before running `airctl create-cluster` or `airctl upgrade`.
### Advanced Commands
| **Command** | **Description** | **Use Case** |
| ---------------- | ----------------------------------------------- | ---------------------- |
| `unconfigure-du` | Delete the Management Plane configuration state | Clean slate deployment |
### Key Configuration Options
#### General Configuration
| **Flag** | **Decription** | **Default** |
| ------------------------------- | --------------------------------------- | --------------------------- |
| `-d, --cluster-deployment-type` | Choose 'onpremk3s' or 'k3s' | onpremk3s |
| `-f, --du-fqdn` | Management plane (DU) FQDN | airctl-1.platform9.localnet |
| `-r, --regions` | Specify regions for the DU (at least 1) | Region1 |
| `-a, --enable-k8s` | Enable the Kubernetes management plane | false |
#### Node Configuration
| **Flag** | **Decription** | **Default** |
| ------------------ | ------------------------------------------ | ----------- |
| `-i, --master-ips` | Master node IP addresses (comma-separated) | - |
| `-w, --worker-ips` | Worker node IP addresses (comma-separated) | - |
| `--ssh-port` int | SSH port for nodes | 22 |
#### Network Configuration
| **Flag** | **Decription** | **Default** |
| -------------------- | ------------------------------------------------------------------------------------------------- | ----------- |
| `-4, --ipv4-enabled` |
Enable IPv4 networking. To enable dual-stack networks, set this flag and "ipv6-enabled"
| true |
| `-6, --ipv6-enabled` |
Enable IPv6 networking. To enable dual-stack networks, set this flag and "ipv4-enabled"
| false |
| `-e, --external-ip4` | Set the external IPv4 for the DU | - |
| `-x, --external-ip6` | Set the external IPv6 for the DU | - |
#### VIP Configuration
| **Flag** | **Decription** | **Default** |
| ---------------------------- | ------------------------------------------------------------ | ----------- |
| `--master-vip4` | Virtual IP for Management Cluster (IPv4) | - |
| `--master-vip6` | Virtual IP for Management Cluster (IPv6) | - |
| `-v, --master-vip-interface` | Specify the masterVipVrouterId since more than 1 master node | - |
| `-z, --master-vip-vrouterid` | Specify the masterVipVrouterId since more than 1 master node | - |
#### Storage Configuration
| **Flag** | **Decription** | **Default** |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `-p, --storage-provider` | StorageProvider (Either 'hostpath-provisioner' or 'custom') (For custom, save provider-specific YAMLs to /opt/pf9/airctl/conf/ddu/storage/custom/) | hostpath-provisioner |
| `-k, --nfs-ip` | Set the NFS server IP address for DU | - |
| `-n, --nfs-share` | Specify the NFS share path for DU | - |
#### K3s Configuration
| **Flag** | **Decription** | **Default** |
| ------------------------------- | -------------------------------------------- | ------------ |
| `-c, --k3s-pod-cidr` | Specify the Pod CIDR for the k3s cluster | 10.20.0.0/16 |
| `-s, --k3s-service-cidr`Service | Specify the service CIDR for the k3s cluster | 10.21.0.0/16 |
#### Registry Configuration
| **Flag** | **Decription** | **Default** |
| ---------------------------------------------------------- | -------------------------------------------------------------------------- | ----------- |
|
--registry-type privateRegistryType
| Type of container image registry to use. Possible values: DU, custom, none | DU |
## Global Flags
These flags can be used with any Airctl command:
| **Flag** | **Decription** | **Example** |
| --------------- | --------------------------------- | ---------------------------- |
| `--config` | config file | `$HOME/airctl-config.yaml` |
| `--quiet` | Disable spinners | |
| `--verbose` | Print verbose logs to the console | `airctl configure --verbose` |
| `-h, --help` | Show command help | `airctl configure --help` |
| `-v, --version` | Version for airctl | |
Configure your management cluster options
Select from various installation types tailored to your specific needs and environmental requirements.
* **Single-master management cluster** - Ideal for proof-of-concept (POC) installations and testing environments where high availability isn't critical.
* **Multi-master management cluster** - Required for production installations where you need high availability and redundancy for your workloads.
## Airctl Configuration Reference
{% tabs %}
{% tab title="YAML" %}
```yaml
# airctl-config.yaml Schema Documentation
# This file documents all possible configuration options for airctl-config.yaml
# Format: YAML with JSON Schema-like descriptions
type: object
properties:
# Installation and Deployment Configuration
installDriver:
type: string
enum: ["DDU", "openstack"]
default: "openstack"
description: "Type of installation driver to use for deployment"
clusterDeploymentType:
type: string
enum: ["onpremk3s", "k3s"]
description: "Type of cluster deployment (onpremk3s or k3s) - k3s is supported in Community Edition"
# DU Configuration
duFqdn:
type: string
required: true
example: "airctl-1.pf9.localnet"
description: "Fully qualified domain name for the DU"
duTenant:
type: string
default: "service"
description: "DU tenant identifier"
duRegion:
type: string
default: "Infra"
example: "Infra RegionOne"
description: "DU region(s) - space-separated for multiple regions"
duUser:
type: string
example: "admin@airctl.localnet"
description: "DU admin user email/username"
duPassword:
type: string
description: "DU admin user password"
# SSH Configuration
sshUser:
type: string
required: true
example: "ubuntu"
description: "SSH username for node access"
sshPort:
type: integer
default: 22
description: "SSH port for node access"
sshPublicKeyFile:
type: string
default: "~/.ssh/id_rsa.pub"
description: "Path to SSH public key file"
sshPrivateKeyFile:
type: string
default: "~/.ssh/id_rsa"
description: "Path to SSH private key file"
# Network Configuration
externalIP:
type: string
example: "10.128.135.5"
description: "External IP (VIP) to access the control plane on the UI. Needs to be in same subnet as cluster nodes"
externalIpV6:
type: string
description: "External IP (VIP) to access the control plane on the UI. Needs to be in same subnet as cluster nodes"
IpV4:
type: boolean
default: true
description: "Enable IPv4 support"
IpV6:
type: boolean
default: false
description: "Enable IPv6 support"
# Storage Configuration
storageProvider:
type: string
enum: ["hostpath-provisioner", "custom"]
default: "hostpath-provisioner"
description: "Storage provider type"
nfsServerIP:
type: string
example: "10.10.10.10"
description: "NFS server IP address"
nfsShare:
type: string
example: "/mnt/pcd-data-share"
description: "NFS share path"
# Installation Paths and Directories
installDir:
type: string
default: "/opt/pf9/airctl"
description: "Airctl installation directory"
logDir:
type: string
default: "~"
description: "Log directory path (defaults to user home)"
# Configuration File Paths
kubeCfgPath:
type: string
default: "/etc/rancher/k3s/k3s.yaml"
description: "Path to Kubernetes configuration file"
bootstrapCfgPath:
type: string
default: "/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml"
example: "/opt/pf9/airctl/conf/k3s-bootstrap-config.yaml"
description: "Path to bootstrap configuration file to deploy the kubernetes cluster"
# Certificate Configuration
clusterFQDN:
type: string
description: "Cluster fully qualified domain name"
caCertPath:
type: string
description: "Path to CA certificate file"
caKeyPath:
type: string
description: "Path to CA key file"
certPath:
type: string
description: "Path to certificate file"
certKeyPath:
type: string
description: "Path to certificate key file"
# Component Version Configuration
consulVersion:
type: string
default: "1.2.0"
description: "Consul version"
vaultVersion:
type: string
default: "0.21.1"
description: "Vault version"
kplaneVersion:
type: string
default: "0.3.5"
description: "Kplane version"
pxcOperatorVersion:
type: string
default: "1.16.1"
description: "Percona XtraDB Cluster Operator version"
pxcDbVersion:
type: string
default: "1.16.1"
description: "Percona XtraDB Cluster Database version"
# Features and Options
enableK8s:
type: boolean
default: true
description: "Enable Kubernetes (K8s/Kaapi) support - default true for On-Prem, false for CE"
analytics:
type: boolean
default: false
description: "Enable Platform9 telemetry and analytics"
pcdChartBundleEnabled:
type: boolean
default: false
description: "Enable PCD chart bundle for airgap mode"
# Timeouts
svcDeploymentTimeout:
type: integer
default: 600
description: "Service deployment timeout in seconds"
nodeShutdownTaintTimeout:
type: integer
default: 3600
description: "Node shutdown taint timeout in seconds"
# Administrative Configuration
adminUserEmail:
type: string
default: "admin@airctl.localnet"
description: "Admin user email address"
# NTP Configuration
ntpServers:
type: string
example: "time.google.com time1.google.com"
description: "Space-separated list of NTP servers for DU time synchronization"
# Options Path
optionsPath:
type: string
default: "/opt/pf9/airctl/conf/options.json"
description: "Path to additional options JSON file"
# Error Collection Configuration (Root Level)
error_collection_enabled:
type: boolean
default: false
description: "Enable error collection and debugging information upload"
error_collection_upload_endpoint:
type: string
example: "https://ce-support-bundle-dev.s3.us-west-1.amazonaws.com"
description: "Endpoint URL for uploading error collection data"
# Additional Notes:
# 1. Required fields vary based on installDriver value (DDU vs openstack)
# 2. For custom storage provider, nfsServerIP and nfsShare are required
# 3. Multiple regions can be specified in duRegion as space-separated values
# 4. Version fields may be automatically determined based on airctl version
```
{% endtab %}
{% endtabs %}
# Enable Kubernetes Management Plane
This section explains how to enable the Kubernetes management plane on an existing Self-Hosted space.vars.product\_name deployment using the `airctl` CLI tool.
## What the Kubernetes Management Plane Is
The Kubernetes management plane runs on the space.vars.product\_name management cluster and lets you provision, upgrade, scale, and manage space.vars.product\_name Kubernetes clusters.
For what space.vars.product\_name Kubernetes provides and how clusters work once this is enabled, see the [space.vars.product\_name Kubernetes Overview](/private-cloud-director/kubernetes-clusters/k8s-overview).
The Kubernetes management plane is optional and opt-in. If you did not enable it during installation (see [Install](/private-cloud-director/getting-started/self-hosted/self-hosted-install)), you can add it later to a running deployment with the steps below.
{% hint style="info" %}
The base management plane must be fully deployed and healthy before you enable the Kubernetes management plane. To confirm, run `airctl status`.
{% endhint %}
## Enable on an Existing Deployment
Run the following command on one of the management cluster hosts to install the Kubernetes management plane on the existing deployment:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl start --k8s-only
```
{% endtab %}
{% endtabs %}
The `--k8s-only` flag installs only the Kubernetes management plane and leaves the rest of the running management plane untouched.
{% hint style="warning" %}
The Kubernetes management plane uses additional resources on the management plane. Ensure your management cluster nodes meet the recommended sizing before enabling it. See [Pre-requisites](/private-cloud-director/getting-started/self-hosted/self-hosted-pre-requisites).
{% endhint %}
## Verify the Installation
Once the command completes, confirm that the Kubernetes management plane is deployed:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl status
```
{% endtab %}
{% endtabs %}
The **Kubernetes management plane** should report `Ready` in the output.
# Adding a New Region to an Existing Deployment
You can add a new region to an already deployed setup using the `airctl`. This allows you to expand your environment without redeploying the entire system.
### Steps to Add a New Region
1. Update the Configuration File:
Edit the `airctl-config.yaml` file and append the new region name under the `duRegion` field. Example:
{% tabs %}
{% tab title="Bash" %}
```bash
# cat /opt/pf9/airctl/conf/airctl-config.yaml | grep -i region
duRegion: Infra RegionOne NewRegion
```
{% endtab %}
{% endtabs %}
2. Run the add-Region Command:
{% tabs %}
{% tab title="Bash" %}
```bash
airctl add-region --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
3. Verify the Region Addition:
Once the command completes, verify that the new region has been successfully added. You should now see the newly added region reflected in both the **state file** and the **`airctl status`** output.
{% tabs %}
{% tab title="Bash" %}
```bash
airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
Here is the sample output.
{% tabs %}
{% tab title="Bash" %}
```bash
------------- deployment details ---------------
fqdn: airctl-1-4108764-426.platform9.localnet
region: airctl-1-4108764-426
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-2561
-------- region service status ----------
desired services: 30
ready services: 30
------------- deployment details ---------------
fqdn: airctl-1-4108764-426-regionone.platform9.localnet
region: airctl-1-4108764-426-regionone
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-2561
-------- region service status ----------
desired services: 77
ready services: 77
------------- deployment details ---------------
fqdn: airctl-1-4108764-426-newregion.platform9.localnet
region: airctl-1-4108764-426-newregion
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-2561
-------- region service status ----------
desired services: 77
ready services: 77
```
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="Bash" %}
```bash
# cat .airctl/state.yaml
...
regions:
- name: Infra
fqdn: airctl-1-4108764-426.platform9.localnet
dbHostname: percona-db-pxc-db-haproxy.foo.svc.cluster.local
dbPassword: *********
- name: RegionOne
fqdn: airctl-1-4108764-426-regionone.platform9.localnet
dbHostname: percona-db-pxc-db-haproxy.foo-regionone.svc.cluster.local
dbPassword: *********
- name: NewRegion
fqdn: airctl-1-4108764-426-newregion.platform9.localnet
dbHostname: percona-db-pxc-db-haproxy.foo-regionone.svc.cluster.local
dbPassword: *********
```
{% endtab %}
{% endtabs %}
This process ensures a seamless addition of new regions to your deployment while maintaining consistency across configuration and runtime state.
# Viewing and Checking Region Status
This section explains how to list all deployed regions in space.vars.product\_name and check their current deployment status using the `airctl` CLI tool.
## List All Deployed Regions
To view the list of all regions currently deployed in the Self-Hosted version of space.vars.product\_name , run:
{% tabs %}
{% tab title="Bash" %}
```bash
/opt/pf9/airctl/airctl get-regions --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
This command retrieves region names directly from the state file and lists all currently active regions. Use it to confirm which regions are available before performing operations such as backups, deletions, or status checks.
Example output:
{% tabs %}
{% tab title="Bash" %}
```bash
$ /opt/pf9/airctl/airctl get-regions --config /opt/pf9/airctl/conf/airctl-config.yaml
Deployed Regions:
• Infra
• LON
```
{% endtab %}
{% endtabs %}
## Check Region Status
The `airctl status` command provides detailed information about the health and service status of each region.
By default, it shows the status of all deployed regions, including service counts and readiness indicators.
### Status Command Flags
The `airctl status` command supports the following flags to customize output:
* \--region : Specifies which region’s status to display. **This flag is mandatory** when using `--ready` or `--desired`.
* \--ready: Displays **only the number of Ready services** for the specified region.
* \--desired: Displays **only the number of Desired services** for the specified region
{% hint style="info" %}
**Info**
* The `--ready` and `--desired` flags are **mutually exclusive** — only one can be used at a time.
* When used, these flags print **only the number of services** in the specified state for the given region.
{% endhint %}
{% tabs %}
{% tab title="Bash" %}
```bash
Usage:
airctl status [flags]
Flags:
--desired Show only the number of desired services for each region
-h, --help help for status
--ready Show only the number of ready services for each region
-r, --region string Display status for a specific region
Global Flags:
--config string config file (default is $HOME/airctl-config.yaml)
--json json output for commands (configure-hosts only currently)
--quiet disable spinners
--verbose print verbose logs to the console
```
{% endtab %}
{% endtabs %}
### Examples:
View All Regions and their Status :
{% tabs %}
{% tab title="Bash" %}
```bash
$ /opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml
------------- deployment details ---------------
fqdn: airctl-1-4161345-238.platform9.localnet
region: airctl-1-4161345-238
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-69
-------- region service status ----------
desired services: 30
ready services: 30
------------- deployment details ---------------
fqdn: airctl-1-4161345-238-lon.platform9.localnet
region: airctl-1-4161345-238-lon
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-69
-------- region service status ----------
desired services: 84
ready services: 84
```
{% endtab %}
{% endtabs %}
View status for a specific region :
{% tabs %}
{% tab title="Bash" %}
```bash
$ /opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml --region Infra
------------- deployment details ---------------
fqdn: airctl-1-4161345-238.platform9.localnet
region: airctl-1-4161345-238
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-69
-------- region service status ----------
desired services: 30
ready services: 30
```
{% endtab %}
{% endtabs %}
View Ready Services for a Region :
{% tabs %}
{% tab title="Bash" %}
```bash
$ /opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml --ready --region LON
84
```
{% endtab %}
{% endtabs %}
View Desired Services for a Region :
{% tabs %}
{% tab title="Bash" %}
```bash
$ /opt/pf9/airctl/airctl status --config /opt/pf9/airctl/conf/airctl-config.yaml --desired --region LON
84
```
{% endtab %}
{% endtabs %}
# Uninstall Self-Hosted Deployment
This section describes how to safely delete a **region** from space.vars.self\_hosted\_product\_name using the `airctl` CLI. Region deletion should be performed carefully, as it involves removing associated configuration and metadata.
### Prerequisites & Safety Checks
Before proceeding with a region deletion, ensure the following:
1. **Take a complete backup** using the steps outlined in the [Backup Guide](https://platform9.com/docs/private-cloud-director/private-cloud-director/backup-and-restore#manual-backup-procedure).
* This ensures that critical region data and configurations can be restored if needed.
2. **Verify the current regions** managed by your setup: `airctl get-regions`
* Review the output and confirm the target region name(s) you plan to delete.
3. Ensure no ongoing deployments, upgrades, or workloads are running in the target region.
4. Gracefully terminate or migrate workloads if needed.
Example :
```bash
$ /opt/pf9/airctl/airctl get-regions --config /opt/pf9/airctl/conf/airctl-config.yaml
Deployed Regions:
• Infra
• LON
```
### Command Syntax
#### Delete a specific region
To delete a specific region, use the following `airctl` command
```bash
airctl unconfigure-du \
--config /opt/pf9/airctl/conf/airctl-config.yaml \
--region
```
This command will uninstall and remove configured regions along with all infrastructure software like consul, vault, percona, k8sniff, etc.
Options:
* `--region ` – Deletes the specified region.
* **Omit the `--region` flag** to delete *all* configured regions.
* Use the `--force` flag if the region deployment was halted or failed midway. This forces cleanup even when the state is inconsistent.
Use the `--force` flag if the region deployment was halted or failed midway. This forces cleanup even when the state is inconsistent.
{% hint style="info" %}
Use the `--force` flag if the region deployment was halted or failed midway. This forces cleanup even when the state is inconsistent.
{% endhint %}
Example:
```bash
airctl unconfigure-du --config /opt/pf9/airctl/conf/airctl-config.yaml --region LON
```
#### Verification & Cleanup
After deletion, verify that the region has been successfully removed. The deleted region should no longer appear in the list of active regions.
```bash
$ /opt/pf9/airctl/airctl get-regions --config /opt/pf9/airctl/conf/airctl-config.yaml
Deployed Regions:
• Infra
```
If you plan to reuse the same nodes to deploy a new self-hosed space.vars.product\_name environment, make sure to also run the following command on all nodes first.
```bash
rm -rf airctl* install-pcd.sh options.json pcd-chart.tgz /opt/pf9/airctl/ .airctl/
```
#### Delete the Management Cluster
The final step is to delete the cluster to complete the cleanup or redeploy from scratch.
```bash
/opt/pf9/airctl/airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml
```
# Airctl Support Bundle
A support bundle is a comprehensive archive created specifically for the purpose of submitting pertinent information to the Platform9 support team.
The support bundle mechanism is designed to be pluggable, allowing for the easy addition of commands, extra logs, or other relevant data to be captured during the archiving process. The steps outlined below will guide you through this procedure.
## Management Plane
For the Management Plane (k3s cluster), the command `airctl gen-support-bundle` is instrumental in generating a comprehensive support bundle. This bundle contains crucial diagnostic information that can be utilized for troubleshooting and support purposes.
Options specific to the `gen-support-bundle` command are:
{% tabs %}
{% tab title="Bash" %}
```bash
Usage:
airctl gen-support-bundle [flags]
Flags:
--all collect logs from all the hosts in the airctl config file
--cluster-dump retrieve cluster-info dump
--cluster-state Get cluster state (default true)
-h, --help help for gen-support-bundle
--host-ips string collect host side logs. Enter unique host ip's in comma-separated format. Example: "10.128.243.113, 10.128.242.112"
--out string the output file name (default "/tmp/supportbundle.tgz")
Global Flags:
--config string config file (default is $HOME/airctl-config.yaml)
--json json output for commands (configure-hosts only currently)
--quiet disable spinners
--verbose print verbose logs to the console
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**Info**
This process will collect logs from all nodes within the management plane cluster, ensuring comprehensive data retrieval for analysis and troubleshooting.
`airctl gen-support-bundle` command should be executed from a Management plane host
{% endhint %}
To generate the support bundle, you can use the following command:
{% tabs %}
{% tab title="Bash" %}
```bash
$ sudo /opt/pf9/airctl/airctl gen-support-bundle --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
By default, the archived contents will be located in the file `/tmp/supportbundle.tgz`. You can share this archived file with the Platform9 support team for further assistance.
To specify a custom path for the archived support bundle, utilize the `--out` option. This allows you to define the exact location where the support bundle will be saved, ensuring that it is easily accessible for future reference or analysis.
{% tabs %}
{% tab title="Bash" %}
```bash
$ sudo /opt/pf9/airctl/airctl gen-support-bundle --out .tgz --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endtab %}
{% endtabs %}
By default, the support bundle collects essential logs from the nodes of the management plane cluster specified in the `bootstrapCfgPath` configuration file.
Support bundle logs includes:
* `dduLogs` : The management plane cluster's (Kubernetes control plane) service logs from the pod/container on the node it's running.
* `/var/log/pf9`: Logs related to the Platform9 services and Kubernetes control plane services running on management plane cluster.
* `kubectl get [pods|nodes|events]` outputs.
* Per-node `airctl-k3s-support.*/` archives — one per management plane node — containing the k3s and containerd logs (`logs/k3s.log`, `logs/containerd.log`, `logs/k3s.journal.log`), k3s systemd unit state (`systemd/k3s.units.txt`, `systemd/k3s.status.txt`), node-local k3s configuration (`config/config.yaml`, `config/k3s.yaml`), and DDU fluentbit logs (`dduLogs/fluentbit/`).
* A `k8s-diagnostics/` directory with a snapshot of Kubernetes API state: `cluster-info.txt`, `events-recent.txt`, `nodes.yaml`, `namespaces.yaml`, `version.yaml`, plus `raw-livez_verbose.txt` and `raw-readyz_verbose.txt` from the `kube-apiserver` health endpoints.
{% hint style="info" %}
**Info**
`-- cluster-dump` option is not functional at the moment.
If the Platform9 support team requests the entire management plane cluster logs, including logs from management plane components, along with the support bundle, please follow the steps below after exporting the KUBECONFIG:
`$ kubectl cluster-info dump --all-namespaces --output-directory=/path/to/dump`
Cluster Dump a set of namespaces
`$ kubectl cluster-info dump --namespaces default,kube-system --output-directory=/path/to/dump`
Next, compress this directory along with the support bundle into a tar archive.
{% endhint %}
## Host Support Bundle
Generate a support bundle for hosts that are connected or authorized to the Management Plane, which we need to troubleshoot the hosts in case of any issues.
Steps for collecting the support bundle from hosts are outlined here: [Hosts Support Bundle](/private-cloud-director/reference/pcdctl-command-line#support-bundle).
# Troubleshooting And Log Files
This section covers log file locations and steps to troubleshoot issues with your self-hosted install.
# Pcddebugger Utility
Learn about PCDdebugger - a command-line Python script designed to simplify and accelerate PCD troubleshooting
## PCDdebugger
PCDdebugger is a command-line Python script designed to simplify and accelerate the troubleshooting process for **PCD Virtualization** environments. It automates the collection of diagnostic information for various services and resources, consolidating the output into a structured directory for easy analysis and sharing.
## Features
* **Comprehensive Data Collection:** Gathers detailed information for key PCD resources including VMs (Nova), images (Glance), networks (Neutron), ports, volumes (Cinder), stacks (Heat), and users (Keystone).
* **Kubernetes Integration:** Performs a complete MySQL dump from a specified Kubernetes namespace, essential for debugging control plane issues.
* **Dependency Traversal:** Automatically discovers and collects data for resources related to a specified VM, such as its ports, volumes, network, subnets, image, and flavor.
* **Organized Output:** Saves all collected data into a timestamped directory, with subfolders for each service, making the information easy to navigate.
* **Archive Option:** Includes a --zip flag to automatically create a compressed archive of the collected data, ready for sharing.
## Prerequisites
Before running the script, ensure the following are installed and configured on your machine:
* **pcd client:** Authenticated and configured to connect to your PCD cloud. (**Ensure your rc file is sourced**).
* **kubectl:** Authenticated and configured to connect to your Kubernetes cluster. (This is required only in case of mysql dump)
## Installation
{% stepper %}
{% step %}
#### Mac-OS
* Download the binary:
{% code title="Command" %}
```bash
curl -LO https://github.com/platform9/PCDDebugger/releases/download/v2.0.1/PCDdebugger-v2.0.1-macos
```
{% endcode %}
* Rename the binary file (Optional):
{% code title="Command" %}
```bash
mv PCDdebugger-v2.0.1-macos PCDdebugger
```
{% endcode %}
* Make the script executable:
{% code title="Command" %}
```bash
chmod +x ./PCDdebugger
```
{% endcode %}
* To run PCDdebugger from any directory, move it to a location in your system's PATH (Optional):
{% code title="Command" %}
```bash
sudo mv PCDdebugger /usr/local/bin/
```
{% endcode %}
{% hint style="info" %}
Mac-OS will ask to allow the permission to run the PCDDebugger binary. Allow it from the System Settings → Privacy and Security.
{% endhint %}
{% endstep %}
{% step %}
#### Linux
* Download the binary:
{% code title="Command" %}
```bash
curl -LO https://github.com/platform9/PCDDebugger/releases/download/v2.0.1/PCDdebugger-v2.0.1-linux
```
{% endcode %}
* Rename the binary file (Optional):
{% code title="Command" %}
```bash
mv PCDdebugger-v2.0.1-linux PCDdebugger
```
{% endcode %}
* Make the binary executable:
{% code title="Command" %}
```bash
chmod +x PCDdebugger
```
{% endcode %}
* To run PCDdebugger from any directory, move it to a location in your system's PATH (Optional):
{% code title="Command" %}
```bash
sudo mv PCDdebugger /usr/local/bin/
```
{% endcode %}
{% endstep %}
{% step %}
#### Windows
* Download the binary:
{% code title="Powershell" %}
```powershell
curl -L https://github.com/platform9/PCDDebugger/releases/download/v2.0.1/PCDdebugger-v2.0.1-windows.exe -o PCDdebugger.exe
```
{% endcode %}
* Place it in a folder (example): Move the downloaded `PCDdebugger.exe` file to a memorable location, for example, C:\Tools.
* (Optional) Add to PATH so you can run it from any command prompt:
1. Search for "Edit the system environment variables" in the Start Menu.
2. Click the "Environment Variables..." button.
3. Under "System variables", find and select the Path variable, then click "Edit...".
4. Click "New" and add the path to the folder (e.g., C:\Tools).
5. Click OK on all windows to save.
* Run `PCDdebugger.exe` from PowerShell or Command Prompt.
{% endstep %}
{% endstepper %}
## Usage
The basic command structure is:
{% tabs %}
{% tab title="command" %}
{% code title="Command" %}
```bash
./PCDdebugger [RESOURCE_FLAG] [OPTIONS] [--insecure]
```
{% endcode %}
{% endtab %}
{% endtabs %}
{% hint style="info" %}
The "--insecure" parameter is mandatory if SSL is not configured; otherwise it will throw an "SSL Certificate" error.
{% endhint %}
### Examples
* Collect all information for a specific VM (gathers the VM, its ports, volumes, network, subnets, image, and flavor):
{% code title="Command" %}
```bash
./PCDdebugger --vm
```
{% endcode %}
* Collect details for a specific Glance image:
{% code title="Command" %}
```bash
./PCDdebugger --image
```
{% endcode %}
* Collect details for a Neutron network and its subnets:
{% code title="Command" %}
```bash
./PCDdebugger --network
```
{% endcode %}
* Perform a MySQL dump from a Kubernetes cluster (requires --namespace):
{% code title="Command" %}
```bash
./PCDdebugger --mysql-dump --namespace
```
{% endcode %}
{% hint style="info" %}
The "--mysql-dump" parameter can only be run on the Self Hosted PCD Virtualization.
{% endhint %}
* Combine multiple flags and create a zip archive:
{% code title="Command" %}
```bash
./PCDdebugger --vm --mysql-dump --namespace --zip
```
{% endcode %}
* Specify a custom output directory and create a zip archive:
{% code title="Command" %}
```bash
./PCDdebugger --vm --output --zip
```
{% endcode %}
Example:
{% code title="Command" %}
```bash
./PCDdebugger --vm aa1bb1cc3-dd4ee5ff6-gg7hh8-ii9jj10kk --output /tmp --zip
```
{% endcode %}
Command-Line Flags:
* `--vm `: Collect details for a specific Nova VM and its related resources.
* `--image `: Collect details for a specific Glance image.
* `--network `: Collect details for a specific Neutron network and its subnets.
* `--port `: Collect details for a specific Neutron port.
* `--volume `: Collect details for a specific Cinder volume.
* `--stack `: Collect details for a specific Heat stack.
* `--user `: Collect details for a specific Keystone user.
* `--mysql-dump`: Perform a MySQL dump of all databases. **Requires** `--namespace`.
* `--namespace `: The Kubernetes namespace to use for the MySQL dump.
* `--output `: Specify a custom directory for the output files.
* `--zip`: Compress the final output directory into a `.zip` file.
* `--help`: Show the help message and exit.
{% hint style="info" %}
All output is saved to a directory named PCDdebugger- by default.
{% endhint %}
## Additional Information
For more details, visit the GitHub page:
# Collecting Cluster Dump For Management Cluster
Learn about pcddump - a powerful utility for collecting comprehensive PCD Management cluster information for offline troubleshooting.
**PCD Dump** is a powerful utility for collecting comprehensive PCD Management cluster information for offline troubleshooting. This script gathers detailed cluster dump information from space.vars.self\_hosted\_product\_name Control Plane, providing administrators with comprehensive diagnostics and troubleshooting data for effective cluster management and issue resolution.
PCD Dump is available here -
## Prerequisites
* Internet connectivity to download the script.
* `curl` installed
* `kubconfig` should be exported and `kubectl` installed with sufficient permissions:
{% tabs %}
{% tab title="Command" %}
{% code title="Verify kubeconfig and internet connectivity" %}
```bash
export KUBECONFIG=
# Verify connectivity
kubectl get nodes
```
{% endcode %}
{% endtab %}
{% endtabs %}
## Installation
{% hint style="warning" %}
Before running the script, ensure that the prerequisites are met.
{% endhint %}
* Run this script to initiate the PCD cluster dump generation:
{% tabs %}
{% tab title="Quick Run" %}
{% code title="One-line quick run" %}
```bash
bash <(curl -Ls https://raw.githubusercontent.com/platform9/PCDDump/refs/heads/main/pcddump.sh)
```
{% endcode %}
{% endtab %}
{% endtabs %}
* For manual execution:
{% tabs %}
{% tab title="Bash" %}
{% code title="Manual download and run" %}
```bash
# Download the script
curl -L https://raw.githubusercontent.com/platform9/PCDDump/refs/heads/main/pcddump.sh -o pcddump.sh
# Make it executable
chmod +x pcddump.sh
# Execute the script
./pcddump.sh
```
{% endcode %}
{% endtab %}
{% endtabs %}
{% hint style="info" %}
The tar output file is saved under /tmp as /tmp/pcddump-$(date +%F\_%H-%M-%S).tar.gz
{% endhint %}
## Additional Information
You can upload the `/tmp/pcddump-$(date +%F_%H-%M-%S).tar.gz` file as suggested by the Platform9 support team. More information in
# Pods Are Displaying FailedAttachVolume or FailedMount Errors
You may run into a problem where the PCD management plane pods are not functioning or running due to PVC mount failures, specifically displaying "FailedAttachVolume" or "FailedMount" errors.
{% tabs %}
{% tab title="Pod Event Section" %}
```ruby
Warning FailedMount 5h32m (x31004 over 6h27m) kubelet MountVolume.WaitForAttach failed for volume "pvc-[pvc-id]" : volume attachment is being deleted
Warning FailedAttachVolume 5h30m (x36 over 6h27m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-[pvc-id]" : volume attachment is being deleted
```
{% endtab %}
{% endtabs %}
## Most common causes
* The storage backend is unreachable.
* The underlying host does not have sufficient resources to run these pods.
* CSI Driver itself is not configured correctly or has some errors.
* Calico network pods are not working as expected.
## Steps to Troubleshoot
{% stepper %}
{% step %}
#### Check pods in init state
Check the number of pods in the `Init` state to identify any pod stuck in initialization. The failure of PVC mount attachments can cause pods to remain in an `Init` state.
{% tabs %}
{% tab title="Command" %}
```bash
$ kubectl get pods -A | grep -i "init"
```
{% endtab %}
{% endtabs %}
{% endstep %}
{% step %}
#### Get CSI drivers
Run the following command to list CSI drivers (storage backend).
{% tabs %}
{% tab title="Command" %}
```bash
$ kubectl get csidrivers
```
{% endtab %}
{% endtabs %}
{% endstep %}
{% step %}
#### Verify CSI driver pods are running
Verify if the CSI driver pods are running. The pods can either be in a dedicated namespace or inside the `kube-system` namespace. In this example the NetApp backend uses the `trident` namespace to host its storage backend pods.
{% tabs %}
{% tab title="Command" %}
```bash
$ kubectl get pods -n /
E.g.
$ kubectl get pods -n trident
trident trident-controller-pod 0/6 ContainerCreating 0 6h44m
trident trident-node-linux-pod 0/2 CrashLoopBackOff 20 (5h33m ago) 23m
trident trident-node-linux-pod 0/2 CrashLoopBackOff 15 (5d4h ago) 23d
trident trident-node-linux-pod 0/2 CrashLoopBackOff 34 (5d3h ago) 23d
```
{% endtab %}
{% endtabs %}
{% endstep %}
{% step %}
#### Inspect Calico pods
As Calico provides pod networking, review all Calico pods and determine why these pods are in a "CrashLoopBackOff/ContainerCreating/OOMKilled/Pending/Error" state; check the events from the describe output.
{% tabs %}
{% tab title="Command" %}
```bash
$ kubectl describe -n
```
{% endtab %}
{% endtabs %}
{% endstep %}
{% step %}
#### Check logs for failing pods
Get more information on the failure from the pod logs.
{% tabs %}
{% tab title="Command" %}
```bash
$ kubectl logs -n /
```
{% endtab %}
{% endtabs %}
{% endstep %}
{% step %}
#### When to escalate
If these steps don't resolve the issue, please contact your Backend Storage Provider or reach out to the Platform9 Support Team for additional assistance.
* Platform9 Support Team:
{% endstep %}
{% endstepper %}
# Delete Orphaned Virtual Machine Entries
When a virtual machine is deleted / migrated / evacuated, sometimes its corresponding allocation on the source hypervisor host is not deleted from the PCD database.
## Root Cause
* If the compute service on the source hypervisor host is stopped abruptly while a VM is being deleted / migrated / evacuated, then the records for the VM in the PCD management database may not be deleted fully.
* As a result, the host-side service cannot communicate with nova-conductor on the controller, so details are not shared and nova-compute may remain under the impression it still holds the virtual machine.
## Resolution
Use the following steps to locate and remove orphaned allocations. Run the commands from a hypervisor host.
{% stepper %}
{% step %}
#### 1. List stale allocations
Run nova-manage placement audit to list allocations for virtual machines that are either deleted or moved to other hypervisor hosts.
{% code title="List stale allocations (run from a pod with nova-manage)" %}
```bash
kubectl exec -it deploy/nova_api_osapi -n -- bash
nova-manage placement audit --verbose
```
{% endcode %}
Verify the output to identify suspected orphaned allocations.
{% endstep %}
{% step %}
#### 2. Verify VM existence / location
Confirm whether the VM still exists or is hosted on another hypervisor host.
{% code title="List servers on source host" %}
```bash
pcdctl server list --all-projects --host
```
{% endcode %}
{% code title="Show instance host" %}
```bash
pcdctl server show -c OS-EXT-SRV-ATTR:host
```
{% endcode %}
{% endstep %}
{% step %}
#### 3. Identify the resource provider ID (if needed)
List resource providers to find the provider associated with the allocation.
{% code title="List resource providers" %}
```bash
pcdctl resource provider list
```
{% endcode %}
{% endstep %}
{% step %}
#### 4. Delete a single orphaned allocation
After validating the allocation is stale and the instance is deleted or moved to another compute node, delete the orphaned allocation.
{% code title="Delete specific allocation" %}
```bash
kubectl exec -it -n -- bash
nova-manage placement audit --verbose --delete
```
{% endcode %}
{% endstep %}
{% step %}
#### 5. Delete multiple/all orphaned allocations
If there are multiple orphaned allocations, delete them all at once.
{% code title="Delete all orphaned allocations" %}
```bash
nova-manage placement audit --verbose --delete
```
{% endcode %}
{% endstep %}
{% step %}
#### 6. Heal allocations
After deletions, run heal\_allocations to ensure Placement entries are consistent.
{% code title="Heal allocations" %}
```bash
nova-manage placement heal_allocations
```
{% endcode %}
{% endstep %}
{% step %}
#### 7. Validate there are no more orphaned allocations
Re-run the audit to ensure no orphaned allocations remain.
{% code title="Validate no orphaned allocations remain" %}
```bash
nova-manage placement audit --verbose
```
{% endcode %}
{% endstep %}
{% endstepper %}
# Using Custom Certificates
Starting from this release, you can configure custom SSL/TLS certificates for your Self-hosted Private Cloud Director deployment. Previously, the system only used self-signed certificates generated during the deployment process.
### Overview
By default, the system generates self-signed certificates during installation. You can now:
* Use your own custom-signed certificates
* Continue using automatically generated self-signed certificates
You can apply custom certificates during a new installation or update existing deployments using the `renew-certs` command.
### Configure Custom Certificates During Installation
Prerequisites
* A valid SSL/TLS certificate file (`.crt`)
* A corresponding private key file (`.key`)
* Read access to both files for the user running the configuration
{% stepper %}
{% step %}
#### Prepare your certificate files
Place your certificate and key files in an accessible location on your system.
Example:
```bash
service.crt # Certificate file
service.key # Private key file
```
{% endstep %}
{% step %}
#### Specify certificate paths
You can provide the certificate and key paths using either of these methods.
Option A: Export as environment variables
```bash
export USER_CERT_PATH="_path_to_service.crt"
export USER_KEY_PATH="_path_to_service.key"
```
Option B: Pass as command-line arguments
```bash
airctl configure {other_flags} \
--user-cert-path _path_to_service.crt \
--user-key-path _path_to_service.key
```
{% endstep %}
{% step %}
#### Verify the configuration
After you run the configuration command, verify that the certificate paths appear in the configuration file.
Check `/opt/pf9/airctl/conf/airctl-config.yaml`:
```yaml
...
user_cert_path: _path_to_service.crt
user_key_path: _path_to_service.key
...
```
Expected outcome: The configuration file contains your specified certificate paths.
{% endstep %}
{% step %}
#### Updating the certs
Run the renew-certs command which will update the certs:
```bash
airctl renew-certs --config _opt_pf9_airctl_conf_airctl-config.yaml
```
{% endstep %}
{% endstepper %}
### Update Certificates on Existing Deployments
If you have an existing PCD deployment, you can replace the current certificates using the `airctl renew-certs` command.
Prerequisites
* You have an existing PCD deployment
* You have the new certificate and key files ready
* You have access to the configuration file at `/opt/pf9/airctl/conf/airctl-config.yaml`
{% stepper %}
{% step %}
#### Update the configuration file
Set the new certificate paths using one of these methods:
* Export environment variables (as shown in the installation section)
* Use the `airctl configure` command with certificate path arguments
* Edit `/opt/pf9/airctl/conf/airctl-config.yaml` directly
> Keep all other configuration fields unchanged.
> {% endstep %}
{% step %}
#### Verify the updated configuration
Check that `/opt/pf9/airctl/conf/airctl-config.yaml` reflects the new certificate paths:
```yaml
certPath: _path_to_new_service.crt
certKeyPath: _path_to_new_service.key
```
{% endstep %}
{% step %}
#### Renew the certificates
Run the certificate renewal command:
```bash
airctl renew-certs --config _opt_pf9_airctl_conf_airctl-config.yaml
```
Expected outcome: The command updates the certificates in your deployment. You can switch from custom to self-signed certificates or from self-signed to custom certificates.
{% endstep %}
{% endstepper %}
{% hint style="info" %}
Important Notes
* Ensure that your `DU_FQDN` environment variable or the `duFqdn` field in the airctl configuration file matches the domain specified in your certificates.
* The user running configuration commands must have read access to the certificate files.
* You can switch between custom and self-signed certificates at any time using the `airctl renew-certs` command.
{% endhint %}
# Teleport Installation
This guide provides comprehensive instructions for installing Teleport agents on Platform9 Private Cloud Director environments.
## Introduction
Teleport enables secure, role-based access to Kubernetes clusters and individual nodes for debugging, monitoring, and management operations. It is a solution for:
* **Secure Remote Access**: SSH and web-based access to cluster resources
* **Role-Based Access Control**: Granular permissions for different user roles
* **Audit and Compliance**: Full logging and session recording
* **Cross-Platform Access**: Unified access to multiple environments
### Prerequisites
* Self-hosted Private cloude Directer is installed and configured
* kubectl access to the **Management cluster**
* Valid Teleport join token from Platform9 Support
#### Installation Options
#### Option 1: Kubernetes Agent Installation
Install Teleport agent into a Kubernetes namespace for namespace-scoped access.
```bash
./install-teleport.sh --token \
--namespace \
--proxy-addr \
--cluster-name \
--version \
--access-namespace \
--access-level \
--resource kube
```
**Parameters:**
* `--token`: Join token (required)
* `--namespace`: Target namespace where the Teleport agent pod runs (default: `teleport`)
* `--proxy-addr`: Teleport proxy address (default: `platform9.teleport.sh:443`)
* `--cluster-name`: name for identification in Teleport interface(default: `example-k8s-cluster`)
* `--version`: Teleport version (default: `18.1.4`)
* `--access-namespace`: Namespace(s) to grant access (comma-separated)
* `--access-level`: Access level - `admin` or `readonly` (default: `admin`)
* `--resource`: Resource type - `kube` (default)
**Examples:**
```bash
# Basic installation with admin access to teleport namespace
./install-teleport.sh --token abc123 --namespace my-app --access-level admin
# Cluster-wide admin access
./install-teleport.sh --token abc123 --access-level admin
# Read-only access to specific namespaces
./install-teleport.sh --token abc123 --access-namespace app1,app2 --access-level readonly
# Custom namespace and cluster name
./install-teleport.sh --token abc123 --namespace production --cluster-name prod-k8s-cluster
```
#### Option 2: Node Installation
Install Teleport node agent on individual Kubernetes nodes for node-level access.
```bash
./install-teleport.sh --token \
--proxy-addr \
--version \
--resource node
```
**Parameters:**
* `--token`: Join token (required)
* `--proxy-addr`: Teleport proxy address (default: `platform9.teleport.sh:443`)
* `--version`: Teleport version (default: `18.1.4`)
* `--resource`: Resource type - `node` (required)
**Example:**
```bash
# Install Teleport on a worker node
./install-teleport.sh --token abc123 --resource node
```
### Access Levels
#### Admin Access
* **Cluster Role**: `cluster-admin` (full cluster access)
* **Namespace Role**: `admin` (full namespace access)
* **Capabilities**: Create, read, update, delete resources
#### Read-Only Access
* **Cluster Role**: `teleport-view-only` (cluster-wide read-only)
* **Namespace Role**: `teleport-namespace-view` (namespace read-only)
* **Capabilities**: Get, list, watch resources only
### What Gets Installed
#### Kubernetes Agent
* **Helm Release**: `teleport-agent` in specified namespace
* **RBAC**: ClusterRole and RoleBindings for specified access level
* **Service Account**: Teleport service account with appropriate permissions
* **Proxy Configuration**: Automatic connection to Platform9 Teleport proxy
#### Node Agent
* **System Service**: Teleport node service installed and started
* **Configuration**: Node configured with join token and proxy settings
* **System Integration**: Integrated with systemd for automatic startup
### Proxy Configuration
The script automatically configures connection to Platform9's Teleport proxy:
* **Default Proxy**: `platform9.teleport.sh:443`
* **Custom Proxy**: Use `--proxy-addr` for alternative proxy
* **Environment Variables**: HTTP\_PROXY, HTTPS\_PROXY, NO\_PROXY automatically configured
### Security Considerations
#### Token Security
* Join tokens are single-use and expire after first use
* Store tokens securely and never commit to version control
* Obtain fresh tokens for each installation
#### Access Control
* **Principle of Least Privilege**: Use read-only access when possible
* **Namespace Scoping**: Limit access to specific namespaces when no cluster-wide access is needed
#### Network Security
* **Proxy Only**: All Teleport connections go through Platform9 proxy
* **TLS Encryption**: All communications encrypted via TLS
* **Firewall Rules**: Ensure outbound connections to proxy are allowed
### Troubleshooting
#### Common Issues
**Token Issues:**
```bash
# Verify token format
echo $JOIN_TOKEN | wc -c # Should be reasonable length
# Check token expiration
# Contact Platform9 Support if token fails
```
**Connection Issues:**
```bash
# Test proxy connectivity
curl -I https://platform9.teleport.sh:443
# Check network connectivity
ping platform9.teleport.sh
```
#### Log Locations
* **Kubernetes Agent**: Check pod logs: `kubectl logs -n teleport-agent-xxx`
* **Node Agent**: Check system logs: `sudo journalctl -u teleport -f`
* **Installation Logs**: Script outputs detailed progress and error messages
#### Cleanup
**Remove Kubernetes Agent:**
```bash
helm uninstall teleport-agent -n
kubectl delete rolebinding,role -n -l app.kubernetes.io/name=teleport
```
**Remove Node Agent:**
```bash
sudo systemctl stop teleport
sudo systemctl disable teleport
sudo rm -f /etc/teleport.yaml
```
### Advanced Configuration
#### Multiple Namespace Access
Grant access to multiple namespaces with comma-separated list:
```bash
./install-teleport.sh --token abc123 \
--access-namespace dev,staging,production \
--access-level readonly
```
This creates RoleBindings in each specified namespace with read-only permissions.
### Support
For issues or questions:
* Contact Platform9 Support
* Verify network connectivity to Teleport proxy
# Availability Zones
Starting from this release, you can configure availability zones for your Private Cloud Director deployment.
### Overview
By default, the nodes were neither labeled nor was the workload distributed across them. With this enhancement, nodes can now be labeled through a YAML configuration, ensuring workloads are scheduled correctly based on geo-location or any other deployment-specific requirements.
This also improves high availability by ensuring that workloads are evenly distributed across all zones during deployment. As a result, even if certain nodes fail, the services remain available and continue operating without disruption.
### Configure AZ's
The AZ's need to be configured before setup creation while running the configure command
```bash
airctl configure \
--master-ips \
--master-vip4 \
--master-vip-interface \
--ipv4 \
--external-ip4 \
--du-fqdn \
--storage-provider \
--node-az-labels /path/to/node-az-labels.yaml
```
The node-az-labels.yaml file should have values in this type
```yaml
# node IP: zone name example
10.10.11.173: asia
10.10.15.162: asia
10.10.15.173: europe
10.10.15.63: europe
10.10.20.11: us
```
Once the above steps are completed, create the cluster. You will then be able to observe that the workload is distributed across the nodes as expected.
# Platform9 OS Beta Installation
{% hint style="warning" %}
**Platform9 OS is currently in closed beta.** Access is by invitation only. To request an invitation or learn more, contact your Platform9 account manager.
{% endhint %}
## Overview
Platform9 OS is powered by Rocky Linux by CIQ (RLC) and space.vars.product\_name. It provides a seamless way to deploy the operating system and space.vars.product\_acronym dependencies in a single workflow using an ISO image. No separate OS installation or manual agent setup is required.
Customers interested in joining the beta program should contact their Platform9 account manager.
## Prerequisites
All prerequisites noted on the [hypervisor configuration prerequisites page](/private-cloud-director/getting-started/pre-requisites#general-hypervisor-configuration-pre-requisites) continue to apply to this installation mode.
## Installation walkthrough
### Step 1: Boot from the ISO
Boot your target server from the Platform9 OS ISO. The Rocky Linux by CIQ (RLC) 10.1 installer screen appears, showing the **Installation Summary** view organized into four categories: **Localization**, **Software**, **System**, and **User Settings**.
Select **Installation Destination** to configure the target disk. On the **Installation Destination** screen, choose the target disk under **Local Standard Disks**. Optionally configure **Storage Configuration** — select **Automatic** for a standard layout or **Custom** to define your own partition scheme. Click **Done** and then **Begin Installation** to start the process.
### Step 2: OS installation and reboot
The installer displays installation progress, including post-installation script execution. When the process completes, a **Reboot System** prompt appears. Allow the system to reboot.
### Step 3: PCD Installer wizard
After the reboot, a four-step text-based installer titled space.vars.product\_name **Installer** launches automatically. Navigate using the keys shown in the footer: `Tab` to navigate between fields, `Space` to edit a selection, `Enter` to confirm, `Esc` to go back, and `Ctrl+C` to exit.
#### Network Configuration (Step 1 of 4)
Configure the network interfaces that this host will use. You can optionally bond multiple interfaces for redundancy using the **Create Bond** action.
If your environment requires an outbound proxy to reach the management plane, enable the **Outbound Proxy** field and enter the proxy address in the format `http://:@:`.
After configuring your network settings, run the built-in network test to verify connectivity before proceeding.
#### Root user password (Step 2 of 4)
Enter and confirm a root user password. This password is used for direct server administration access.
#### GPU Configuration (Step 3 of 4)
This step is optional. If the host has GPUs installed, you can configure them now. The following options are available:
* **Skip GPU configuration** — do not configure GPUs during this installation.
* **PCI Passthrough** — pass discrete GPUs directly to guest VMs for exclusive, bare-metal GPU access.
* **vGPU pre-configure** — prepare the host for NVIDIA vGPU host driver setup.
* **vGPU SR-IOV configure** — configure SR-IOV vGPU mode for hardware-partitioned GPU sharing.
* **Validate Passthrough** — verify that a previously configured passthrough setup is functioning correctly.
* **Validate vGPU** — verify that a previously configured vGPU setup is functioning correctly.
{% hint style="info" %}
For in-depth details on GPU Passthrough and vGPU setup, configuration options, and supported GPU models, see [GPU Support in space.vars.product\_name](/private-cloud-director/gpu/gpu-support-pcd).
{% endhint %}
#### PCD Connection Details (Step 4 of 4)
Enter the management plane connection details so the host can register with your space.vars.product\_acronym instance:
* **Account URL:** the URL of your space.vars.product\_acronym management plane.
* **Username:** your space.vars.product\_acronym account username.
* **Password:** your space.vars.product\_acronym account password.
* **Region** and **Tenant** details as applicable to your environment.
You can find these values in the space.vars.product\_acronym UI under **Infrastructure > Cluster hosts > Add New Hosts**.
{% hint style="info" %}
If you are setting up a traditional software-only self-hosted management plane rather than connecting to an existing one, see [Self-Hosted Install](/private-cloud-director/getting-started/self-hosted/self-hosted-install) for that path.
{% endhint %}
## Post-install Host Boot Console
Once space.vars.product\_acronym components are installed and the host has registered, the screen presents the space.vars.product\_name **— Host Boot Console**. This console displays the host specifications:
* **CPU** — processor model and core count (for example, "8 x 12th Gen Intel(R) Core(TM) i9-12900KS").
* **Memory** — total available memory (for example, "14.99Gi").
* **Network** — the primary interface name and its assigned IP address (for example, "ens3 — 172.16.122.88/24").
Two footer actions are available: press `F2` to open the **System Configuration** menu, or press `F12` to drop to a shell.
### System Configuration menu (F2)
Pressing `F2` opens the **System Configuration** menu with two options:
* **Configure Management Network** — update network settings after the initial installation.
* **View System Service Status** — open the service status view to inspect and manage space.vars.product\_acronym daemons.
### Viewing and restarting services
The **View System Service Status** screen lists each space.vars.product\_acronym-managed daemon with a status indicator. The services shown include **Network**, **Comms**, **Hostagent**, and **Sidekick**. Each service displays its current status (for example, **Active**).
If a service is not running, select it and use the **Restart** action to bring it back up without rebooting the host.
# Accept Certificate Authority
In the current release of space.vars.product\_name , a few services are configured with self-signed certificate. You must trust the certificate by accepting the self-signed certificate for proper operation of these services.
Following services require you to trust the certificate:
1. Image Library Service - You need to trust the certificate before you can upload any Images using space.vars.product\_name UI or API.
1. When using the OpenStack CLI, you can use `--insecure` option to upload images to the Image Library Service without trusting the certificate.
2. Kubernetes Service - You need to trust the cluster endpoint certificate for each Kubernetes cluster that you create, before you can view any cluster specific data in the space.vars.product\_name UI.
To accept the certificate:
1. Click on the link in the UI where you see a notification that says "Action Required: Trust Certificate"
2. The link will open in a new tab. Expand on the "Advance" option in the tab and proceed to click on the link to trust the certificate.
3. Note that in some cases you may need to change the security settings for your browser before you can trust the certificate.
# Host Operating System Support Matrix
This page describes which host operating systems are supported for each Private Cloud Director release, along with deprecation timelines and end-of-support dates.
| PCD Release | Ubuntu 22.04 | Ubuntu 24.04 | Platform9 OS powered by Rocky Linux from CIQ (RLC) 10.2 |
| ---------------- | -------------- | ------------ | ------------------------------------------------------- |
| **January 2026** | Supported | Supported | -- |
| **April 2026** | Supported | Supported | -- |
| **August 2026** | Deprecated | Supported | Supported |
| **October 2026** | Deprecated | Supported | Supported |
| **January 2027** | Deprecated | Supported | Supported |
| **April 2027** | End of support | Supported | Supported |
**Legend:**
* Supported = fully tested and maintained
* Deprecated = still functional, but no longer recommended for new deployments; plan to migrate
* End of support = no longer tested, patched, or supported
{% hint style="info" %}
**Ubuntu Support**
Ubuntu host operating systems follow a bring-your-own-OS model. You provide, install, and maintain Ubuntu yourself, and Platform9 supports Private Cloud Director on the Ubuntu versions marked Supported in the table above. Platform9 does not package, harden, or manage the Ubuntu operating system, and there is no Platform9-delivered Ubuntu image equivalent to Platform9 OS.
{% endhint %}
{% hint style="info" %}
**Rocky Linux Support**
Private Cloud Director is supported only on Platform9 OS powered by Rocky Linux from CIQ (RLC) 10.2 and installed through the Platform9 OS ISO. Platform9 OS is a controlled, hardened configuration: it is hardened to the CIS Level 1 benchmark and is configured to pull packages and updates only from the Platform9 validated package repository. Platform9 builds, tests, patches, and supports this operating system together with PCD as a single integrated stack.
Community Rocky Linux is not a supported PCD host operating system. This applies regardless of the version number: a host running community Rocky Linux 10.2 is still unsupported, because it does not carry the Platform9 OS hardening and does not use the Platform9 validated package repository, so it is not the configuration PCD is built and tested against. Private Cloud Director installed on a customer-provided Rocky Linux host is an unsupported configuration.
{% endhint %}
### Important dates
**August 2026:** Ubuntu 22.04 is deprecated. New cluster deployments should use Ubuntu 24.04 or Platform9 OS powered by Rocky Linux 10.2.
**January 2027:** Last PCD release that includes Ubuntu 22.04 support.
**April 2027:** Ubuntu 22.04 reaches end of support. Hosts running Ubuntu 22.04 must be migrated before upgrading to this release.
### Migration between host operating systems
Workload migration support depends on whether the source and destination hosts run the same host operating system.
| Migration type | Between Ubuntu and Platform9 OS hosts | Between Platform9 OS hosts |
| -------------- | ------------------------------------- | -------------------------- |
| Live migration | Not supported | Supported |
| Cold migration | Supported | Supported |
For details on migrating workloads from Ubuntu hosts to Platform9 OS hosts, see [Cross-OS Cold Migration: Ubuntu Hosts to Platform9 OS Hosts](/private-cloud-director/virtualized-clusters/virtualmachine/vm-migration#cross-os-cold-migration-ubuntu-hosts-to-platform9-os-hosts).
# Configuration Maximums
This page lists the configuration limits Platform9 has validated for Private Cloud Director (PCD). Stay at or below these values to ensure full Platform9 support.
**These are tested limits, not the architectural maximums of the underlying components.** The individual components often support higher values. If your requirements exceed the published values, contact Platform9 support for guidance.
Actual achievable limits depend on hardware selection, workload characteristics, and network topology, and it may not be possible to hit every maximum simultaneously. Consult the individual subsystem sections (compute, network, storage) when sizing your environment. To scale beyond the per-Region maximums, deploy multiple Regions.
## Virtual Machine Maximums
Limits that apply to a single VM running on PCD.
### Storage
Criteria
Value
Virtual disks per VM (root + attached volumes)
8
Boot volume size
4TB
Ephemeral disk size
No hard maximums exist
### Networking
Criteria
Value
Security groups per port
10
Security group rules per security group
100
Floating IPs per VM
50
## Hypervisor Host Maximums
Limits that apply to a single PCD hypervisor host.
### Host Compute
Criteria
Value
Maximum vCPU oversubscription ratio
16×
Maximum RAM oversubscription ratio
~1.5×
### Region Maximums
Limits that apply to a single PCD Region (the management plane controlling a set of hypervisors). Customers requiring scale beyond these values should plan for multiple Regions.
Criteria
Value
Clusters per Region
8
Hypervisors per Region
591 active; 629 total registered
VMs per Region
9,673 powered on VMs
Tenants per Region
74
Users per Region
136
Host Aggregates per Region
9
Flavors per Region
15
Images per Region
7
Concurrent VM create operations per Region
50
Concurrent VM migrate operations per Region
5
## High Availability (VM HA) and DRR Maximums
Limits that apply to PCD VM HA clusters.
Criteria
Value
Hypervisors per cluster
100
## Networking Maximums
Limits that apply to the software-defined networking layer.
### Networks and Subnets
Criteria
Value
Tenant networks per Region
14
Subnets per Region
13
Subnets per network
100
Ports per Region
19,992
Ports per network
10,000
### Addressing and Isolation
Criteria
Value
Security groups per Region
450
VLAN ID range per physical network
-
VXLAN / Geneve tunnel endpoints per Region
591
### Layer 2 Networks
Criteria
Value
Ports per Layer 2 network
500
## Block Storage Maximums
Limits that apply to the block storage service.
Criteria
Value
Volumes per Region
10,000
Volumes per tenant
10,000
Backends per Region
6
Volume types per Region
6
## Tenant Maximums
Default quotas applied to each tenant.
Criteria
Value
VMs per tenant
10,000
vCPUs per tenant
10,000
RAM per tenant
8,000 GB
Volumes per tenant
10,000
Volume storage per tenant
10,000 GB
Networks per tenant
100
Routers per tenant
10
Floating IPs per tenant
50
Security groups per tenant
500
Key pairs per user
100
# Upgrade Overview
This section covers upgrading a space.vars.product\_name environment — preparing your hosts, sequencing the upgrade safely, and verifying that every service is healthy afterward.
space.vars.product\_name supports two deployment models:
* **SaaS** — Platform9 hosts and operates the management plane.
* **Self-Hosted** — you operate the management plane on-premise.
The runbooks below apply to both models. Steps that are specific to the on-premise (Self-Hosted) management plane — primarily the `airctl` commands — are called out separately within each runbook.
## In this section
* [Host Upgrade Runbook](/private-cloud-director/upgrade/host-upgrade-runbook) — pre-upgrade environment checks, host upgrade ordering across multi-role hosts, Ubuntu 22.04-to-24.04 OS upgrade caveats, and failure recovery.
* [Post-Upgrade Verification](/private-cloud-director/upgrade/post-upgrade-verification) — the checklist to confirm region health, host roles, GPU, storage, and networking are healthy, then re-enable VM HA and DRR.
## Related pages
* **Self-Hosted management plane upgrade:** [Upgrade Management Plane and Hosts](/private-cloud-director/getting-started/self-hosted/upgrade) covers upgrading an on-premise (Self-Hosted) deployment with `airctl`.
# Host Upgrade Runbook
## Overview
This runbook guides you through upgrading the hosts in a space.vars.product\_name virtualized cluster. It covers the pre-upgrade environment checks you must complete before upgrading any host, the recommended ordering when hosts carry multiple roles, Ubuntu 22.04-to-24.04 host OS upgrade considerations, and how to recover from common upgrade failures.
{% hint style="info" %}
**Applies to both deployment models.** space.vars.product\_name supports two deployment models — **SaaS** (Platform9 hosts and operates the management plane) and **Self-Hosted** (you operate the management plane on-premise). The host-level guidance on this page applies to both models. Steps that act on the on-premise management plane — for example, the `airctl` commands and management-plane health checks — apply to **Self-Hosted deployments only** and are called out in boxes like the ones below. In SaaS deployments, Platform9 operates and upgrades the management plane.
{% endhint %}
In this guide, you will prepare your environment for a host upgrade, sequence the upgrade safely across multi-role hosts, handle Ubuntu OS upgrades without losing network connectivity, and recover if an upgrade stops partway through.
## Pre-Upgrade Checklist
Complete every item in this checklist before upgrading any host. Skipping checks is the most common cause of upgrade failures and extended maintenance windows.
### Check Region Health
The management plane must be healthy before any host upgrade begins. A degraded management plane cannot coordinate the host upgrade process reliably.
From the space.vars.product\_name UI, navigate to **Infrastructure > Regions** and confirm that all regions show a healthy status. Resolve any failing services before continuing.
{% hint style="info" %}
**Self-Hosted deployments only.** From the management cluster node, confirm region health from the command line:
```bash
airctl status
```
Confirm that every region shows `region health: ✅ Ready` and that `desired services` matches `ready services`. Resolve any failing pods or services before continuing.
{% endhint %}
### Verify Per-Host Free Disk Space
Insufficient disk space on a host causes the upgrade package installation to fail partway through, leaving the host in a partially upgraded state that is difficult to recover.
SSH to each host you plan to upgrade and check the following mount points:
```bash
df -h /var /var/log /opt /tmp
```
Recommended minimums before starting a host upgrade:
| Mount point | Recommended free space |
| ----------- | ---------------------- |
| `/var` | At least 5 GB |
| `/var/log` | At least 2 GB |
| `/opt` | At least 3 GB |
| `/tmp` | At least 1 GB |
If `/var/log` is nearly full, rotate or archive old logs before proceeding:
```bash
journalctl --vacuum-size=1G
find /var/log -name "*.gz" -mtime +30 -delete
```
### Confirm All Hosts Are Authorized and Healthy
Every host you intend to upgrade must show an `applied` role status and an `online` connection status. Hosts in `failed`, `error`, `converging`, or `unknown` states should be resolved before the upgrade.
From the UI, navigate to **Infrastructure > Cluster Hosts** and confirm that no host shows a warning or error badge.
{% hint style="info" %}
**Self-Hosted deployments only.** You can also check host status from the management cluster node:
```bash
airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
Look for any host where `Status` is not `ok` or `Agent Status` is not `running`. Investigate and resolve those hosts first.
{% endhint %}
### Verify Storage Backend Connectivity
If any host carries the Persistent Storage Service role (block storage) or the Image Library Service role, verify that those services are reachable and healthy before upgrading those hosts.
**Persistent Storage Service (block storage):**
```bash
pcdctl volume service list
```
All volume service endpoints should show `enabled` and `up`. If any endpoint is `down` or `disabled`, investigate before upgrading the host that carries that role.
**Image Library Service:**
```bash
pcdctl image-service list
```
Confirm that the Image Library Service endpoints are `enabled` and `up`.
You can also verify storage and image library health from the UI: navigate to **Infrastructure > Storage** and **Infrastructure > Image Library** respectively.
### Verify Network Connectivity
Confirm that all hosts are reachable over the management network. From any host (or, for Self-Hosted deployments, the management cluster node) that can reach the management network:
```bash
for ip in ...; do
ping -c 2 "$ip" && echo "$ip OK" || echo "$ip UNREACHABLE"
done
```
Replace ``, ``, and so on with the management IP addresses of your hosts. If any host is unreachable, resolve the connectivity issue before upgrading.
### Disable VM HA and DRR Before Host OS Upgrades
{% hint style="warning" %}
**Important: disable VM HA and DRR before upgrading host operating systems.**
VM evacuation and live migration require matching operating system and KVM versions between source and destination hosts. If VM HA or DRR is active while some hosts run Ubuntu 22.04 and others run Ubuntu 24.04, evacuation attempts between those hosts will fail.
{% endhint %}
If you are upgrading the host OS (for example, from Ubuntu 22.04 to 24.04) — as distinct from upgrading the space.vars.product\_name host agent packages only — disable VM HA and DRR at the cluster level before starting:
1. Navigate to **Infrastructure > Clusters** in the UI.
2. Select the cluster whose hosts you are upgrading.
3. Toggle **VM High Availability** to off.
4. Toggle **Dynamic Resource Rebalancing** to off.
Re-enable both only after all hosts in the cluster have been upgraded to the same OS version. See [Post-Upgrade Verification](/private-cloud-director/upgrade/post-upgrade-verification#re-enable-vm-ha-and-drr) for the re-enablement steps.
For full pre-conditions and behavior details, see [Virtual Machine High Availability](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha) and [Dynamic Resource Rebalancing (DRR)](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr).
### Plan Your Maintenance Window
Estimate the upgrade window before you begin:
* Host agent-only upgrades (no OS change) typically complete in 5–10 minutes per host.
* Host OS upgrades (Ubuntu 22.04 to 24.04) require an additional 20–40 minutes per host plus reboot time.
Schedule the window to avoid peak workload hours. Communicate the window to application owners, because workloads on hosts entering maintenance mode will be live-migrated to other cluster hosts during the upgrade.
If any VMs remain on a host while its OVN controller is upgraded — for example, when maintenance mode is skipped or VM HA is disabled — those VMs briefly lose network packets while the controller restarts and re-establishes flows. Platform9 testing has observed roughly **4–6 seconds** of packet loss per host (4 seconds with 47 VMs on Ubuntu 24.04, 6 seconds with 15 VMs on Ubuntu 22.04). Existing TCP connections recover automatically. Following the maintenance-mode procedure below avoids this disruption by draining the host before its OVN controller restarts.
## Host Upgrade Ordering and Role Dependencies
When hosts carry multiple roles, the order in which you upgrade them affects service availability. Follow these sequencing rules. They apply to both deployment models.
### Recommended Upgrade Order
1. **Hosts with only the Hypervisor role** — upgrade these first. They carry no shared service responsibility, so their upgrade has the narrowest blast radius.
2. **Hosts with the Networking Service role** — upgrade these next. Put each host into maintenance mode before upgrading to migrate its VMs first.
3. **Hosts with the Image Library Service role** — upgrade image library hosts one at a time. Confirm the Image Library Service is healthy on the remaining hosts before upgrading the next one.
4. **Hosts with the Persistent Storage Service role** — upgrade block storage hosts one at a time. Confirm that volumes are accessible from other hosts before proceeding to the next block storage host.
5. **Multi-role hosts** — hosts that carry hypervisor, image library, or block storage roles simultaneously are the highest-risk to upgrade. Treat them like the most sensitive single role they carry (block storage > image library > hypervisor-only).
### Put Each Host Into Maintenance Mode Before Upgrading
Before upgrading any host, use maintenance mode to drain its VMs:
1. Navigate to **Infrastructure > Cluster Hosts**.
2. Select the host.
3. Click **Other > Enable Maintenance Mode** and follow the prompts.
4. Wait for the **Migration Status** banner to show all VMs migrated.
Then proceed with the upgrade for that host. After the upgrade is complete and the host is back online, disable maintenance mode before moving to the next host.
See [Maintenance Mode](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/maintenance-mode) for full details on the enable and disable process.
### Persistent Storage Service Host Considerations
If your region has only one host with the Persistent Storage Service role, upgrading that host briefly makes block storage unavailable. Schedule that upgrade during a maintenance window when no block storage volume attach or detach operations are expected. If possible, add a second block storage host before the upgrade window so there is no single point of failure.
### Image Library Service Host Considerations
If your region has only one host with the Image Library Service role, new VM provisioning from images will fail during that host's upgrade. All existing running VMs continue operating normally. Schedule that upgrade during a window when no new VMs need to be deployed.
## Ubuntu 22.04 to 24.04 Host OS Upgrade Caveats
{% hint style="warning" %}
**Network connectivity risk on hypervisor hosts.**
Upgrading from Ubuntu 22.04 to 24.04 on a host with the Hypervisor role carries a risk of OVS (Open vSwitch) bridge misconfiguration during the OS upgrade. If the OVS bridge loses its interface binding after the reboot, the host loses network connectivity and cannot re-connect to the management plane. Plan for out-of-band console access (IPMI, iDRAC, or equivalent) before starting the OS upgrade on hypervisor hosts.
{% endhint %}
### Before the OS Upgrade
1. Record the current OVS bridge configuration on the host:
```bash
ovs-vsctl show
ip addr show
```
Save the output. You will need it to verify the configuration is intact after the reboot.
2. Confirm the host is in maintenance mode and all its VMs have been migrated off.
3. If the host has the Hypervisor role, ensure you have out-of-band console access in case network connectivity is lost after the OS upgrade.
### After the OS Upgrade and Reboot
Perform network re-validation immediately after the host comes back online:
1. Confirm the OVS bridge is present and the physical interface is still a member:
```bash
ovs-vsctl show
```
The output should show your bridge (for example, `br-ex` or `br-data`) with the physical interface listed as a port. If the physical interface is missing from the bridge, re-add it:
```bash
ovs-vsctl add-port
```
Replace `` and `` with the values from your pre-upgrade recording.
2. Confirm the host can reach the management plane:
```bash
curl -sk https:///resmgr/v1/hosts | head -c 200
```
3. Confirm the host agent is running and connected:
```bash
systemctl status pf9-hostagent
```
The service should be `active (running)`. If it is stopped or failed, restart it:
```bash
systemctl restart pf9-hostagent
```
4. From the management plane, confirm the host shows `online` connection status in **Infrastructure > Cluster Hosts**.
5. After confirming network and host agent health, disable maintenance mode and allow the host to rejoin scheduling before proceeding to the next host.
### Mixed OS Versions in a Cluster
Do not leave a cluster in a mixed OS state longer than necessary. A cluster where some hosts run Ubuntu 22.04 and others run Ubuntu 24.04 has the following limitations:
* VM HA evacuations between mixed-OS hosts will fail (KVM version mismatch).
* DRR live migrations between mixed-OS hosts will fail.
* Manual VM migrations between mixed-OS hosts are not supported.
Keep VM HA and DRR disabled for the entire cluster until all hosts are on the same OS version.
## Host Upgrade Failure Recovery
The host agent and package-level recovery steps below apply to both deployment models. Steps that re-submit an upgrade through `airctl` or inspect the region's Kubernetes namespace apply to Self-Hosted deployments only and are marked. In SaaS deployments, if a host upgrade fails and the host-level steps below do not recover it, contact Platform9 Support.
### 409 Conflict Error During Host Upgrade
A `409 Conflict` response during a host upgrade typically means the management plane has a stale lock or an in-progress record for that host from a previous attempt. This prevents the upgrade from being re-submitted.
**Recovery steps:**
1. Verify the host agent is running and reports back to the management plane:
```bash
# On the affected host
systemctl status pf9-hostagent
```
{% hint style="info" %}
**Self-Hosted deployments only.** Check whether the host upgrade pod from the previous attempt is still running, and clear it before retrying:
```bash
kubectl get pods -n | grep host-upgrade
```
If a `host-upgrade-*` pod is in `Running` or `Pending` state from a prior attempt, wait for it to complete or delete it:
```bash
kubectl delete pod -n
```
{% endhint %}
Once the host agent is confirmed running (and, for Self-Hosted deployments, the stale pod is cleared), retry the host upgrade. If the conflict persists, contact Platform9 Support.
### "Cluster Name Removed from Host" Error
This error appears when the host's local configuration no longer contains the cluster association record. It typically occurs if the host was deauthorized or had its roles removed while an upgrade was in progress.
**Recovery steps:**
1. Check the host's role status. From the UI, navigate to **Infrastructure > Cluster Hosts** and inspect the affected host.
2. If the host shows `unauthorized` or missing roles, re-authorize it and re-assign roles from the UI: select the host, click **Edit Roles**, and re-assign the appropriate roles.
3. Wait for the host to reach `applied` role status (the status transitions through `converging`).
4. Once the host is back to `applied` status, retry the host upgrade.
### Partial or Failed Upgrade Cleanup
If a host upgrade fails midway, the host may be in an inconsistent state with mixed package versions. Before retrying, perform the following cleanup on the affected host:
1. Check which space.vars.product\_name packages are installed on the host and their versions:
```bash
dpkg -l | grep pf9
```
2. If you see a mix of old and new package versions, force-reinstall the current target packages to bring the host to a consistent state:
```bash
apt-get install --reinstall pf9-ostackhost pf9-neutron-base pf9-neutron-ovn-controller
```
Add any other `pf9-*` packages shown by the `dpkg -l` output that are at the wrong version.
3. Restart the host agent after reinstalling:
```bash
systemctl restart pf9-hostagent
```
4. Wait for the host to return to `applied` status in the management plane, then retry the upgrade.
### Host Re-Sync and Retry
If a host is stuck in `converging` state for more than 10 minutes after a failed upgrade:
1. From the UI, navigate to **Infrastructure > Cluster Hosts**, select the host, and use **Other > Re-sync Host** (if available) to trigger a re-convergence.
2. Alternatively, restart the host agent on the host directly:
```bash
systemctl restart pf9-hostagent
```
3. Monitor the hostagent log for convergence progress:
```bash
tail -f /var/log/pf9/hostagent.log
```
Look for `converge successful` or `role application complete` messages.
4. If the host does not converge within another 10 minutes, contact Platform9 Support with the hostagent log from the affected host (and, for Self-Hosted deployments, the output of `airctl host-status`).
## Next Steps
After all hosts are upgraded, complete the [Post-Upgrade Verification](/private-cloud-director/upgrade/post-upgrade-verification) checklist to confirm that all services are healthy and re-enable VM HA and DRR.
# Post-Upgrade Verification
## Overview
After upgrading all hosts, run through this verification checklist to confirm that every service is healthy and every role is operating correctly. Do not re-enable VM HA or DRR until you have completed the service and role verification steps below.
{% hint style="info" %}
**Applies to both deployment models.** The host, role, storage, networking, and GPU verification steps on this page apply to both **SaaS** and **Self-Hosted** deployments. Management-plane verification using `airctl` applies to **Self-Hosted deployments only** and is marked. In SaaS deployments, Platform9 operates and upgrades the management plane; verify region health from the UI.
{% endhint %}
In this guide, you will confirm region health, verify all host roles, validate GPU passthrough configuration, test storage volume attach, test Networking Service connectivity, and re-enable VM HA and DRR.
## Verify Region Health
From the space.vars.product\_name UI, navigate to **Infrastructure > Regions** and confirm all regions show a healthy status before proceeding.
{% hint style="info" %}
**Self-Hosted deployments only.** Run `airctl status` from the management cluster node and confirm every region is ready:
```bash
airctl status
```
Expected output for each region:
```
deployment status: ready
region health: ✅ Ready
desired services:
ready services:
```
The `desired services` and `ready services` counts must match. If any services are not ready, check the pod logs in the region namespace before proceeding:
```bash
kubectl get pods -n | grep -v Running | grep -v Completed
```
Investigate any pod in `CrashLoopBackOff`, `Error`, or `Pending` state.
{% endhint %}
## Verify Host Role Status
Every host in every region must show `Status: ok` and `Agent Status: running` after the upgrade.
From the UI, navigate to **Infrastructure > Cluster Hosts** and confirm:
* No host shows a warning or error badge.
* All hosts that had the Hypervisor role before the upgrade still show the Hypervisor role as `applied`.
* Hosts with the Persistent Storage Service role show that role as `applied`.
* Hosts with the Image Library Service role show that role as `applied`.
If a host's role is missing or shows as `unauthorized` after the upgrade, re-assign the role: select the host, click **Edit Roles**, assign the appropriate roles, and click **Update Role Assignment**. If a host is stuck in `converging`, select the host and choose **Other > Re-sync Host**.
For any host that is not healthy, check the host agent log on that host:
```bash
tail -n 100 /var/log/pf9/hostagent.log
```
{% hint style="info" %}
**Self-Hosted deployments only.** You can also verify host role status from the management cluster node:
```bash
airctl host-status --config /opt/pf9/airctl/conf/airctl-config.yaml
```
{% endhint %}
## Validate GPU Passthrough
{% hint style="info" %}
Skip this section if your deployment does not use GPU passthrough.
{% endhint %}
After upgrading hosts that carry the Hypervisor role with GPU passthrough configured, re-validate that the GPU devices are still bound correctly.
1. On each GPU host, confirm the GPU device is bound to the `vfio-pci` driver:
```bash
lspci -k | grep -A 3 -i nvidia
```
The `Kernel driver in use` field should show `vfio-pci`. If it shows a different driver, the GPU binding was not preserved through the OS upgrade. Follow the [Set up GPU Passthrough](/private-cloud-director/gpu/gpu-support-pcd/set-up-gpu-passthrough) guide to rebind the device.
2. From the space.vars.product\_name UI, navigate to **Infrastructure > Cluster Hosts**, select a GPU host, and confirm that the GPU devices are listed under the host's hardware details.
3. Attempt to launch a test VM using a GPU-enabled flavor. Confirm the VM starts successfully and the GPU is accessible from within the VM.
For vGPU deployments, confirm that the vGPU profiles are still present and the vGPU driver service is running on the host:
```bash
systemctl status nvidia-vgpu-mgr
```
For full GPU troubleshooting steps, see [Troubleshooting GPU Support](/private-cloud-director/gpu/gpu-support-pcd/troubleshooting-gpu-support).
## Test Persistent Storage Volume Attach
Confirm that the Persistent Storage Service is operational by attaching a volume to a running VM.
1. List available volumes:
```bash
pcdctl volume list
```
2. Identify a test VM that is running:
```bash
pcdctl server list --status ACTIVE
```
3. Attach an existing available volume to the test VM:
```bash
pcdctl server add volume
```
4. Confirm the attachment succeeded:
```bash
pcdctl server volume list
```
The volume should appear with a `in-use` status. Detach the test volume after confirming:
```bash
pcdctl server remove volume
```
If the attach fails, check the Persistent Storage Service endpoint status:
```bash
pcdctl volume service list
```
All endpoints should be `enabled` and `up`. If an endpoint is `down`, check the volume service logs on the block storage host.
## Test Networking Service Connectivity
Verify that the Networking Service is healthy and that VM network connectivity is intact after the upgrade.
### Verify Networking Service Endpoints
```bash
pcdctl network agent list
```
All agents should show `alive: True`. If any agent is not alive, check the networking agent status on the affected host:
```bash
systemctl status pf9-neutron-ovn-controller
```
### Test Network and Router Connectivity
1. List the networks in the region and confirm the expected networks are present:
```bash
pcdctl network list
```
2. List routers and confirm they are in `ACTIVE` state:
```bash
pcdctl router list
```
3. From a test VM, confirm outbound connectivity (if the VM has network egress):
```bash
# From within the test VM
ping -c 4 8.8.8.8
```
If any router is not `ACTIVE`, navigate to **Infrastructure > Networking > Routers** in the UI and inspect the router details for error messages.
### Test VM-to-VM Connectivity
Confirm that VMs on different hosts can communicate across the virtual network:
1. Identify two running VMs on different hypervisor hosts.
2. From one VM, ping the private IP address of the other:
```bash
ping -c 4
```
If ping fails between VMs on different hosts, check the OVS bridge status on the hypervisor hosts:
```bash
ovs-vsctl show
ovs-ofctl dump-flows br-int | head -20
```
## Re-Enable VM HA and DRR
Re-enable VM HA and DRR only after:
* All hosts in the cluster are running the same operating system version.
* All hosts show `Status: ok` and all roles are `applied`.
* The region health check passed.
* Storage and networking tests passed.
{% hint style="warning" %}
Do not re-enable VM HA if any hosts in the cluster are still running a different OS version. VM evacuation between hosts with different KVM versions will fail.
{% endhint %}
**Re-enable VM HA:**
1. Navigate to **Infrastructure > Clusters**.
2. Select the cluster.
3. Toggle **VM High Availability** to on.
4. Confirm the cluster shows `Protected` VM HA status after enabling. If it shows `Degraded` or `Not Protected`, hover over the VM HA status to see which prerequisite is not met, and resolve it before relying on VM HA for workload protection.
**Re-enable DRR:**
1. Navigate to **Infrastructure > Clusters**.
2. Select the cluster.
3. Toggle **Dynamic Resource Rebalancing** to on and confirm the frequency setting is correct for your environment.
For pre-conditions and full behavior details, see [Virtual Machine High Availability](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha) and [Dynamic Resource Rebalancing (DRR)](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr).
## Upgrade Complete
Your space.vars.product\_name upgrade is complete once all verification steps above have passed and VM HA and DRR are re-enabled with healthy statuses. If you encounter issues during verification that you cannot resolve, contact Platform9 Support with the relevant pod or service logs. For Self-Hosted deployments, also include the output of `airctl status` and `airctl host-status`.
# Cluster Blueprint
A **Virtualized Cluster** in space.vars.product\_name is a group of hypervisor hosts that virtual machines get provisioned on.
You can create one or more virtualized clusters per region. You can also further slice a cluster into multiple subgroups of hosts using [Host Aggregates](/private-cloud-director/virtualized-clusters/host-aggregate). This allows you to combine hosts with similar characteristics into a cluster or a host aggregate within a cluster and target VM provisioning to that cluster or aggregate.
## What is Cluster Blueprint
A cluster blueprint allow you to describe common configure that all virtualized clusters will share in a declarative, prescriptive manner. Blueprint is designed to help you express your desired cluster architecture upfront, and ensure that cluster capacity that is added over time conforms to this desired architecture.
## Create a Cluster Blueprint
To deploy and use a **virtualized cluster**, your first step will be to create a cluster blueprint.
Navigate to Infrastructure -> Cluster Blueprints in the space.vars.product\_name UI to create your cluster blueprint.
Follow the details in [Networking Configuration](/private-cloud-director/virtualized-networking/networking-overview#networking-configuration) to find out more about configuring networking service as part of the cluster blueprint configuration.
Follow the details in [Block Storage Service Configuration](/private-cloud-director/storage/block-storage#block-storage-service-configuration) to learn about creating one or more storage types as part of cluster blueprint.
### Customize Cluster Defaults
Here you get to customize storage location for image library and virtual machine ephemeral storage.
### Image Library Storage Location
Specify the storage type and location where virtual machine images will be stored.
You can specify a **file system path or a volume type name** for this option.
If you specify a **volume type name** for the image library location, that volume type will be used to store images in the image library. If you specify a filesystem path instead, you must indicate whether the path corresponds to **local** or **shared storage**. Enable the **Shared Storage** toggle if the path is configured to use NFS or an equivalent setup that is shared and mounted on all image library hosts.
### Virtual Machine Storage Path
The virtual machine storage path specifies the disk location on each hypervisor that will be used to store:
* Ephemeral root disk files for VMs running on this hypervisor that are using ephemeral disks.
* Any additional VM metadata and files.
Read [Ephemeral Storage](/private-cloud-director/storage/ephemeral-storage#what-is-ephemeral-storage) for an in-depth understanding of what is ephemeral storage and it's advantages and disadvantages before you configure this for your setup.
The default value for this path is `/opt/data/instances`. You can modify the default to a different location based on your preference.
This can be a local disk or an NFS shared disk, depending on how the storage path is configured at each hypervisor level.
Note that you configure this path once at the blueprint level for all hosts across all clusters in your hypervisor.
### Virtual Machine Console IP
Specify an IP address or a domain name to be used to access the console of all VMs in the region. You can configure this in the blueprint. Ensure that proper routing or DNS resolution is in place so the console can be accessed reliably. Read more [details here](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint/setting-up-a-console-proxy) on how to set this up.
## Update Cluster Blueprint
Some configurations in the cluster blueprint may be updated after hosts are added to the clusters, others are subject to limitations depending on how they are being applied to existing hosts. Cluster Network Parameters and Image Library and VM storage locations cannot be edited once hosts are authorized to the space.vars.product\_acronym region.
You can still make changes to cluster blueprint to add a network interface to an existing in-use Host Configuration or update the key-value metadata for Volume Backend Configurations. These configuration changes do get propagated to existing hosts that are already part of a cluster. When adding a new network interface to an in-use Host Configuration, you would not be able to change the system traffic options to use this new interface, but you could create a new Physical Network using the Physical Network Label associated with this interface.
{% hint style="info" %}
**Editing Host Config**
Within the **Edit Roles** modal, the **Host Config** (network configuration) selection can only be modified when no roles are currently installed on the host. If roles (such as Hypervisor, Image Library, or Persistent Storage) are already active, the network configuration dropdown will be disabled. To change the network configuration for an existing host, the host must be fully reset. This process requires the removal of all assigned roles from the host within the blueprint, followed by a **Deauthorize** and **Decommission** action on the host. Once the host is cleared and successfully re-onboarded to the platform, the **Host Config** can be modified during the role assignment process.
{% endhint %}
# Setting Up a Console Proxy
space.vars.product\_name allows you to proxy all of the VM console connections in a region through one or more servers. You can configure this in the cluster blueprint using the field **VNC Proxy IP or Domain Name,** and set it to a domain name or an IP address. The proxy must always point to a hypervisor node in the same region. Using any other server or VM is not supported.
#### How console proxy traffic works
When a user opens a VM console, the connection does not go directly to the hypervisor running the VM. Every console session in the region first reaches the hypervisor that the console proxy endpoint resolves to (the **authoritative node),** and that host then proxies the session to the hypervisor that actually runs the VM.
In the example below, the endpoint resolves to **Hyp1**, which acts as the VNC proxy and forwards the session to **Hyp2**, where the VM runs. **Hyp3** is not involved in this session.
```
┌────────────────┐
│ User / │
│ Browser │
└────────┬───────┘
│ VNC console request
▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Hyp1 │ │ Hyp2 │ │ Hyp3 │
│ (VNC proxy node) │ │ │ │ │
│ │ │ ┌──────────┐ │ │ │
│ │─────▶│ │ VM │ │ │ │
│ │ │ └──────────┘ │ │ │
└──────────────────┘ └──────────────────┘ └──────────────────┘
Hyp1 is the resolved host and acts as the VNC proxy. It forwards the
session to Hyp2, which runs the requested VM. Hyp3 and any other
hosts can stay isolated from direct user access.
```
1. **Request.** The user opens a VM console from the UI or API. space.vars.product\_name returns a console URL that points to the console proxy endpoint — the IP or domain name set in the blueprint — on port `6080`, together with a one-time token that identifies the target VM and the hypervisor running it.
2. **Resolve the endpoint to a hypervisor.** The console proxy endpoint always resolves to a hypervisor in the region (never a standalone server or VM). Which host becomes the authoritative node depends on how you configured the proxy:
* **A hypervisor's IP** — every session is routed through that one fixed host.
* **A floating IP / VIP** — the session reaches whichever host currently holds the VIP (for example, managed by keepalived).
* **A domain name** — the name can map to one or more hypervisors. See [Using a domain name](#using-a-domain-name) below.
3. **Forward to the correct hypervisor.** The noVNC proxy service on the authoritative node uses the token to identify the hypervisor running the target VM, opens a backend connection to it over the internal host-to-host network, and relays the console stream back to the user.
Only the authoritative node needs to be reachable by users. Every other hypervisor can have external access blocked, as long as hypervisor-to-hypervisor traffic on the console port is allowed.
**Using a domain name**
A domain name can map to more than one hypervisor. In that case your DNS, VPN, or `/etc/hosts` decides which host the request lands on, and that host then proxies the session to the hypervisor running the VM. In the example below the name resolves to either **Hyp1** or **Hyp3**, and from there the session reaches **Hyp2**:
```
┌──────────────────┐
│ User / │
│ Browser │
└─────────┬────────┘
│ console.example.com
│ (DNS resolves to Hyp1 or Hyp3)
┌─────────────────────┴─────────────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Hyp1 (VNC proxy) │ │ Hyp3 (VNC proxy) │
└─────────┬────────┘ └─────────┬────────┘
│ │
└─────────────────────┬─────────────────────┘
▼
┌──────────────────┐
│ Hyp2 │
│ │
│ ┌──────────┐ │
│ │ VM │ │
│ └──────────┘ │
└──────────────────┘
The domain name maps to more than one hypervisor (here Hyp1 and Hyp3).
Your DNS, VPN, or /etc/hosts decides which one the request lands on;
that host then proxies the session to Hyp2, which runs the VM.
```
#### Options for configuring the console proxy
**1. Use a hypervisor’s IP address**
If you have multiple hypervisors, you can route all console connections through a single host by setting that hypervisor’s IP as the proxy IP in the blueprint. This allows you to restrict public access to all other hypervisors.
**2. Use a public /** **floating IP**
If your hypervisor nodes can route a public or floating IP, you can set that floating IP in the blueprint. Traffic will then flow through whichever host currently holds the floating IP. Tools like keepalived can be used to manage this.
**3. Use a domain name**
You can also provide a domain name instead of an IP. To make this work, map the domain name to one or more hypervisor hosts either:
* locally in your `/etc/hosts`, or
* through your organization’s DNS or VPN configuration.
#### Limitations
The console proxy is not a standalone, externally hosted gateway. It must be one (or more) of the hypervisors in the same region. Using any other server or VM is not supported. As a result:
* At least one hypervisor must remain reachable by users to act as the proxy (the jump point). A configuration where *all* hypervisors are isolated from direct user access, with only an external proxy exposed, is not supported today.
* The remaining hypervisors can be fully isolated from direct user access. They only need to accept console (host-to-host) traffic from the authoritative node.
# Virtualized Cluster
space.vars.product\_name enables you to manage and operate multiple virtualized clusters from a single space.vars.product\_acronym region.
## Use Cases
1. **Multi-Tenant Isolation** - You can assign different tenants to separate clusters to enforce resource boundaries and fault domain separation. The isolation prevents noisy neighbor issues and enhances security between tenant environments.
2. **Licensing Requirements -** Some specialized software, such as Oracle database, may require isolating hosts with the software license enabled. Creating a separate cluster will allow you to do this.
3. **Hardware Specialization -** You can group hosts with similar capabilities (such as GPU-enabled or high-memory machines) into dedicated clusters to optimize performance and resource utilization for specialized workloads.
## Understanding a Virtualized Cluster
A virtualized cluster is a grouping of interconnected physical servers or hosts, with certain cluster level features operating at the group level. Resources such as CPU, Memory, and GPU across all hosts in a virtualized cluster are presented as a single pool, so that when you create a new virtual machine in the virtualized cluster, the placement of the new virtual machine may be on any of the underlying physical servers, based on capacity and other provisioning constraints.
All clusters within a region operate under a **single cluster blueprint**, ensuring consistency in base configurations while allowing for cluster-specific customization.
Each virtualized cluster provides two key features that enable you to run production workloads.
### Create a Virtualized Cluster
To create a new virtualized cluster:
1. Navigate to **Infrastructure** > **Clusters** > **Add Cluster**
2. Provide a name for the cluster.
3. Choose desired settings for:
1. VMHA (Virtual Machine High Availability)
2. DRR (Dynamic Resource Rebalancing)
3. GPU (Passthrough or vGPU)
### CPU Mode and Model
Defines how the hypervisor exposes CPU features to virtual machines. You can select one of the following CPU modes during cluster creation in the space.vars.product\_name UI.
* **Default:** Automatically selects the latest supported CPU model on the host.
* **Host Model:** Matches the physical CPU closely, balancing performance and migration flexibility. This mode is recommended for clusters with mostly homogeneous hardware (e.g., all Intel Skylake or all AMD EPYC of similar generation).
* **Host Passthrough:** Exposes the physical CPU directly for maximum performance (migration only possible between identical hosts). Host passthrough is not generally recommended due to migration limitations. This mode is suitable for:
* Clusters with effectively identical host CPUs.
* Clusters where the best possible performance is desired without any live migration needs.
* **Custom**: Specify a standardized CPU model to ensure consistent virtualized CPUs across hosts and enable live migration between different hardware. This option lets you pick a CPU model that matches the oldest or least capable CPUs in your fleet, allowing live migration between hosts even if they span several CPU generations. The trade-off is that you cannot fully leverage the features and performance of your newer CPU in favor of live migration compatibility.
### Virtual Machine High Availability (VM HA)
VM HA provides automatic fault tolerance for your workloads. When enabled:
* The system continuously monitors host health across the cluster.
* If a host failure is detected, affected VMs are automatically recovered on healthy hosts.
* Minimize downtime without requiring manual intervention.
* Business continuity is maintained even during infrastructure issues.
Read more here about [Virtual Machine High Availability (VM HA)](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha).
### Dynamic Resource Rebalancing (DRR)
DRR works as a continuous optimization engine that:
* Monitors allocated capacity and real-time utilization metrics (CPU and memory) across all hosts in the cluster.
* Analyzes resource distribution patterns to identify imbalances
* Intelligently migrates VMs across hosts in the cluster to optimize resource utilization.
* Prevents hotspots and resource contention before they impact performance
This proactive approach ensures that your clusters maintain optimal performance even as workload patterns change over time.
Read more here about [Dynamic Resource Rebalancing (DRR)](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr).
### GPU
Enabling GPU support allows you to group all hosts capable of GPU passthrough or vGPU-backed hypervisors into a single GPU-aware virtual cluster.
Before configuring GPU mode on any GPU hosts, ensure the following points are met.
* Supported GPU Modes: Passthrough or vGPU
* Use Passthrough to assign full physical GPUs to virtual machines.
* Use virtual GPUs (vGPUs) to virtually slice existing physical GPUs and share them across virtual machines.
* Non-GPU hosts can not be added to a GPU-enabled cluster.
This setting ensures that all GPUs in the cluster use the same GPU mode (Passthrough or vGPU) for uninterrupted GPU operations, such as live migration between GPU hardware, resizing, and scaling GPUs.
Read more here about [GPU Support in Private Cloud Director](/private-cloud-director/gpu/gpu-support-pcd).
# Dynamic Resource Rebalancing (DRR)
In this document you will learn about space.vars.product\_name Dynamic Resource Rebalancing (DRR), an automated feature that optimizes resource utilization of your virtualized clusters. DRR ensures efficient resource utilization via live migration of virtual machines, preventing performance bottlenecks.
## Introduction
In any dynamic virtualized environment, workloads fluctuate. Some virtual machines might become resource-intensive, while others remain idle. This can lead to imbalances across the physical hypervisor hosts within a cluster – some hosts become overloaded, causing performance degradation for their VMs, while other hosts sit underutilized. space.vars.product\_name addresses this challenge with its integrated **Dynamic Resource Rebalancing (DRR)** feature. DRR works continuously to ensure that resources like CPU and memory are utilized efficiently and that no single host becomes a performance bottleneck.
## Benefits of DRR
Enabling DRR in your space.vars.product\_name environment delivers tangible benefits:
* **Consistent VM Performance:** Helps prevent performance issues caused by resource contention on overloaded hosts.
* **Optimized Resource Utilization:** Ensures that your hardware investment is used more efficiently across the cluster.
* **Proactive Problem Avoidance:** Identifies and resolves potential resource bottlenecks before they negatively impact applications.
* **Reduced Operational Overhead:** Automates the complex task of monitoring and balancing VM workloads, freeing up administrator time.
## DRR Pre-requisites
* DRR requires at **least two hosts** in a virtualized cluster to operate.
* DRR has the same pre-requisites as [VM Live Migration Prerequisites](/private-cloud-director/virtualized-clusters/virtualmachine/vm-migration#live-migration-prerequisites).
* We recommend that you configure the cluster hosts to have the same [CPU and Memory over-commitment ratio](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster#host-overcommitment-allocation-ratios) best performance.
* Make sure to read the current [#known-issues-and-limitations](#known-issues-and-limitations "mention") section before proceeding to enable DRR on a cluster.
## Configure DRR for a Cluster
* **Scope** - DRR is configured as a property at **each virtualized cluster level**. When you create a new virtualized cluster, you have the option to enable DRR for the cluster.
* You can also enable DRR on an existing virtualized cluster at a later point.
* **Frequency** - When you enable DRR, you are asked to choose a **value for frequency** at which DRR should run.
* The default is **20 minutes**. Which means every 20 minutes, DRR will evaluate hosts within the virtualized cluster and identify targets for potential rebalancing.
* You can change this value to **10 minutes or 30 minutes**.
## DRR and VM Migration Priority
### What Is Migration Priority
Migration priority is a key concept that is part of space.vars.product\_name DRR. Migration priority can be assigned at per virtual machine level. It determines:
* whether DRR should migrate a virtual machine and, if so,
* the priority in which it should be selected for migration.
### Migration Priority Values
Migration priority supports following values:
* **Unset** - This is the default state for migration priority value for virtual machines.
* **Normal** - Any virtual machines that do not have migration priority explicitly assigned, and that do not have soft affinity rules associated with them, will be treated with migration priority normal. The VMs are chosen for migration after the VMs with `high` priority value are migrated, but before the VMs with `low` priority value are migrated.
* **Low** - VMs with a priority value of `low` will be **selected** **last** for migration. You may therefore assign this value to virtual machines that incur a heavier tax on migration and that you would only want to live migrate if all other options are exhausted. VMs with soft affinity rules that do not have migration priority explicitly assigned will also be treated with low priority.
* **High** - VMs with a priority value of `high` will be **selected** **first** for migration. A good candidate for this category are dev/test VMs that may be smaller in size and / or their performance may not be impacted by live migration.
* **Excluded** - You can also set a migration priority value of `never` to a VM. VMs with this value will be not be migrated by the DRR service. You can use this for virtual machines that you do not want DRR to ever migrate.
**Note** that the migration priority values are only relevant in the context of DRR. These values are not taken into account by other space.vars.product\_name services such as [Virtual Machine High Availability](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha) that may still evacuate the VM to a different host, if enabled at cluster level. An Administrator can also manually migrate a VM at any point independent of the migration priority value set for that VM.
{% hint style="warning" %}
**Important**
Migration priority values are only relevant in the context of DRR. They are not honored by other space.vars.product\_name services such as VM HA. Administrators can manually migrate a VM independent of the migration priority.
{% endhint %}
### Configure Migration Priority For VMs
You can change the migration priority value for any VM running on a virtualized cluster with DRR enabled.
* Select the VM in the virtual machines grid view by navigating to the 'virtual machines' menu in the space.vars.product\_name UI.
* From the action bar, choose 'Other actions', then choose 'Migration Priority'.
* You can now select the appropriate priority value for the VM.
## How DRR Works
DRR functions as an ongoing optimization engine for your cluster:
1. **Continuous Monitoring:** DRR continuously monitors key resource utilization metrics, specifically CPU and memory utilization, across all active hosts within the virtualized cluster.
2. **Imbalance Detection:** The system analyzes CPU and Memory utilization to identify imbalances in resource utilization across hosts.
1. DRR looks for hosts that are overloaded in the cluster.
2. **Host overload threshold**: When DRR finds a host with **CPU OR memory utilization of greater than 80%**, it determines that the host is overloaded.
3. DRR then ensures that there are underutilized hosts with sufficient spare capacity in the cluster.
4. Assuming it finds the required spare capacity, DRR will identify a suitable target host for migration. DRR will sort all compatible hosts in the cluster based on their utilization value, and **choose the host with lowest utilization**.
5. DRR takes into account per host [overcommitment or over-allocation ratio](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster#host-overcommitment-allocation-ratios) while finding a suitable candidate host.
6. DRR then initiates VM migrations from overloaded hosts to other compatible hosts with spare capacity.
3. **Automated Live Migration:** DRR first groups VMs from the source host based on the VM Migration Priority.
1. VMs with no priority value assigned to them are treated to have priority value of `default`
2. VMs with priority value of `high` are chosen for migration first
3. VMs with priority value of `default` are chosen next
4. VMs with priority value of `low` are chosen last.
5. VMs with priority value of `never` are skipped from migration.
6. DRR will then initiate VM live migrations, **one VM at a time**. This ensures that there is *no downtime* for the virtual machine while it is being migrated.
7. After each VM live migration, DRR re-evaluates if the source host is still overloaded. If true, it will continue with this process.
## DRR Interoperation with other services
This section describes how DRR interoperates with other services configured for your cluster.
### Host Aggregates
DRR only finds candidate target hosts for a VM that satisfy the specific VM's host aggregate requirement. If it does not find a suitable host, DRR will not migrate the VM.
### VM HA
DRR and Virtual Machine High Availability are designed to interoperate well together. It is possible that a host failure event might occur while DRR is actively rebalancing VMs, either from the same host or from other hosts in the cluster. When this happens:
1. VM HA will detect the host failure and initiate VM evacuations
2. The VM evacuations may result in cluster imbalance
3. DRR will then detect the imbalance during either the current run or the next run.
4. DRR will redistribute the load across the cluster to address the imbalance.
### VMs with Hard Affinity or Anti-Affinity Rules
1. **DRR will skip a VM with hard affinity rule** from migration.
1. This behavior may change in the future once DRR has ability to initiate bulk VM migrations.
2. For a VM with hard anti-affinity rule, DRR will find suitable candidate host that satisfies the anti-affinity rule. If it can not find a suitable host, DRR will not migrate the VM.
For a VM with soft-affinity, DRR will try its best to find a target host that satisfies affinity, if not it will find another target host to migrate the VM. Also if migration priority is not set for the VM, DRR will consider the priority as low for the VM
For a VM with soft-anti-affinity, DRR will try its best to find a target host that satisfies anti-affinity, if not it will find another target host to migrate the VM. Also if migration priority is not set for the VM, DRR will consider the priority as low for the VM
### VM States
DRR **only live migrates VMs in Active or powered on state**. DRR will skip VMs in all other states.
### VMs with Special Properties
1. DRR will live migrate VMs that have hot added CPU or memory resources.
### Host States
DRR will **only operate on hosts in online state**. DRR will ignore hosts in offline or error state.
DRR will **only operate on hosts with hypervisor role assigned**. DRR will ignore all other hosts.
## Known Issues and Limitations
* DRR does not presently have the ability to live migrate VMs with [Virtual TPM](/private-cloud-director/virtualized-clusters/virtual-tpm) enabled today. This ability is coming soon.
* DRR will skip a VM with hard affinity rule from migration.
* This behavior may change in the future once DRR has ability to initiate bulk VM migrations.
* For a VM with hard anti-affinity rule, DRR will find suitable candidate host that satisfies the anti-affinity rule. If it can not find a suitable host, DRR will not migrate the VM.
# Virtual Machine High Availability (VM HA)
In this document, you will learn about space.vars.product\_name **Virtual Machine High Availability (VM HA)**, a feature that automatically detects physical host failures within a cluster and restarts the affected VMs on other healthy hosts in the same cluster.
## Introduction
Hardware, firmware, or network issues can cause a virtualization host to go offline with little warning. Without an automated recovery mechanism, every VM on that host would remain down until an operator intervenes, violating most service-level objectives.
**Virtual Machine High Availability (VM HA)** protects against this risk. The process is designed to be automatic and requires minimal manual intervention during a failure event:
* Continuous Host Monitoring: The VM HA service continuously monitors the health and responsiveness of all hypervisor hosts in an HA-enabled virtualized cluster.
* Failure Detection: If a host stops responding (due to hardware failure, operating system crash, or certain network isolation scenarios), the system detects the failure
* Automatic VM Recovery: Upon confirmation of a host failure, which involves verifying the failure on both the management plane and the cluster hosts, VM HA automatically restarts any VMs running on the failed host. These VMs are powered on using available resources on the remaining healthy hosts within the cluster.
After recovery, complementary features such as **Dynamic Resource Rebalancing (DRR)** can redistribute load to restore optimal balance across the cluster, ensuring sustained performance.
## Benefits of VM HA
Enabling VM HA in your Private Cloud Director environment delivers the following benefits:
* **Minimized Downtime**: Automatically restarts or evacuates VMs when a host fails, reducing mean time to recovery from hours to minutes.
* **Service Continuity**: Keeps business-critical applications online even during unexpected infrastructure outages.
* **Operational Efficiency**: Eliminates the need for round-the-clock manual monitoring and intervention, freeing administrators to focus on higher-value tasks.
* **Policy-Driven Control**: Respects host aggregates, affinity/anti-affinity rules, and VM-specific settings, enabling you to determine how each workload is handled during failover.
* **Seamless Interoperation**: Works in concert with DRR and other space.vars.product\_name services to maintain both availability and resource efficiency after a failure event.
## VM HA Pre-Requisites
VM HA always operates within a virtualized cluster. You need to turn it on at the cluster level. Once enabled, VM HA applies to all virtual machines in the cluster.
* Shared storage is required for VM HA. [Shared storage requirements for VM HA](https://platform9.com/kb/pcd/storage/shared-storage-requirements-for-vmha) details the effect of various shared storage configurations on VM HA performance. For VM HA to work at the cluster level:
* All VMs should be using a block storage volume as the root disk (non-ephemeral root disk), or
* If any VMs use ephemeral storage for the root disk, [Ephemeral Shared Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-shared-storage) should be used for all hosts in the virtualized cluster, and the **This is Shared Storage** toggle is enabled in the **Cluster Blueprint** under the **Customize Cluster Defaults** tab.
* When [Ephemeral Shared Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-shared-storage) is not used, any VMs using an ephemeral root disk will be rebuilt on another host during recovery.
* VM HA requires a minimum number of healthy hosts in a cluster to function correctly. A minimum of two hosts is required for HA activation.
* VM HA uses the VM Evacuation operation behind the scenes. VM Evacuation Prerequisites must be met for the operation to succeed.
* If any VMs in the cluster use a [Flavor](/private-cloud-director/virtualized-clusters/virtualmachine/vm-flavors) that assigns the VM to a host aggregate, then that host aggregate should have at least two hosts in the cluster for VM HA failover redundancy.
* If any VMs in the cluster use block storage, the block storage role must be assigned to at least two hosts in the cluster.
* If any VMs in the cluster use vTPM, then
* The [virtual machine ephemeral storage directory](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint#virtual-machine-storage-path) and the [vTPM state file directory](/private-cloud-director/virtualized-clusters/virtual-tpm#vtpm-state-file) must be on shared storage (eg NFS) that is mounted on all of your hypervisor hosts in the cluster.
* This is required as long as you are using vTPM as a feature for your VMs, even if the VMs do not otherwise use ephemeral storage for their root disk.
* These directories must be owned by `pf9` user and `pf9group` group.
* The Image library role must be assigned to at least two hosts in the cluster, and the Image library must use shared storage.
* **Operating System Compatibility**: All hosts in the VM HA-enabled cluster must run the same operating system version.
* Mixed Ubuntu versions (22.04 and 24.04) in the same cluster are not supported.
* VM evacuation between different OS versions will fail due to the KVM version incompatibilities.
* We recommend disabling VM HA before upgrading the operating system on any host of the cluster, then re-enabling it after the host(s) have been upgraded.
* Remember to read and understand the current VM HA before proceeding to configure it at the cluster level.
## Configure VM HA for a Cluster
VM HA configuration follows a two-step process to ensure cluster readiness:
#### Step 1: Create the Cluster
When you create a new cluster via **Infrastructure > Clusters > Add Cluster**, the **VM High Availability** toggle will appear but be disabled. You cannot enable VM HA during cluster creation.
#### Step 2: Enable VM HA After Adding Hosts
After creating the cluster:
1. Add at least two hosts with the hypervisor role to the cluster.
2. Verify that all hosts are assigned to the same host aggregate (if using host aggregates)
3. Navigate to the cluster settings and enable the **VM High Availability** toggle
{% hint style="info" %}
**NOTE**
The VM HA toggle is disabled until the minimum requirements are met. Hovering over the disabled toggle displays the message: "At least two hosts are required to enable VM HA."
{% endhint %}
### VM HA Enablement Rules
The system enforces these validation rules for VM HA:
* **Minimum Host Requirement**: At least 2 hypervisor hosts are required in the cluster.
{% hint style="info" %}
If fewer than two hosts are present, the toggle displays: *"At least two hosts are required to enable VM HA."*
{% endhint %}
* **Host Aggregate Consistency**: All hosts must either belong to the same host aggregate or have no aggregate assignment.
{% hint style="info" %}
If the hosts are part of different aggregates, the toggle displays the following message:*"VM HA cannot be enabled because hosts belong to different host aggregates, which prevents cross-host migration."*
{% endhint %}
On the **Clusters** grid view, you will see one of the following VM HA statuses for each cluster:
* **Disabled**: Shows the number of clusters for which VM HA is disabled.
* **Protected**: Shows the number of clusters with VM HA enabled that meet all prerequisites.
* **Degraded**: Shows the number of clusters with VM HA enabled that have a single point of failure (for example, only one host in the cluster has the persistent storage or image library role). Such clusters can absorb host failures as long as the host is not the single point of failure.
* **Not Protected**: Shows the number of clusters with VM HA enabled that have not met one or more prerequisites.
### Upgrading your Cluster Hosts
When upgrading your cluster hosts from one operating system version to another (for example, Ubuntu 22.04 to 24.04), follow the steps below:
* **Disable VM HA** before starting the upgrade process.
* **Evacuate all VMs** from the hosts that are scheduled for upgrade.
* **Upgrade hosts** one at a time to the new operating system version.
* **Do not attempt** to migrate VMs between upgraded and non-upgraded hosts.
* **Re-enable VM HA** only after all hosts have been upgraded to the same operating system version.
{% hint style="warning" %}
**Important**
VM HA-driven evacuation and live migration require matching operating system and KVM versions between source and destination hosts. Attempting to live migrate or evacuate VMs between Ubuntu 22.04 and 24.04 hosts, or between Ubuntu and Platform9 OS powered by Rocky Linux by CIQ hosts, will fail. **Cold migration** from Ubuntu to Platform9 OS hosts *is* supported; see [Cross-OS Cold Migration: Ubuntu Hosts to Platform9 OS Hosts](/private-cloud-director/virtualized-clusters/virtualmachine/vm-migration#cross-os-cold-migration-ubuntu-hosts-to-platform9-os-hosts).
{% endhint %}
## How VM HA Works
space.vars.product\_name **Virtual Machine High Availability (VM HA)** provides automated protection against host failures by continuously monitoring host health and, when needed, evacuating affected virtual machines (VMs) to healthy hosts with minimal disruption. Three cooperating services enable the automated workflow:
1. **High Availability Manager**: Central coordinator that runs in the space.vars.product\_name management plane and watches for clusters with VHMA enabled, gathers health reports, confirms host failures, and issues evacuation requests.
2. **Host Agents: Daemons that run on every hypervisor host and probe their peers’ liveness and report findings to the High Availability Manager at regular intervals.**
3. **VM Evacuation Service:** Orchestrator that runs in the space.vars.product\_name management plane and receives confirmed *host‑down* events from **High Availability Manager** and migrates each impacted VM to a suitable target host.
### End-to-end Flow
1. **Cluster discovery:** The **High Availability Manager** polls the infrastructure to discover clusters where VHMA is enabled. For VM HA-enabled clusters, it verifies that the pf9-ha-slave role is present on every hypervisor host. Clusters with fewer than two hosts are ignored until additional hosts join.
2. **Peer list distribution:** Every 127 seconds, each Host Agent requests a *peer list* from the **High Availability Manager**. The list is a randomized subset of hosts in the same cluster, helping to spread traffic and avoid single‑point bias.
3. **Distributed health probing:** Using the peer list, the host agent performs a liveness check on its peer hosts via the libvirt exporter. This check is entirely agent‑to‑agent, with no central polling path.
4. **Status aggregation & reporting:** Every 59 seconds, the agent posts an aggregated view of peer health to the **High Availability Manager**.
5. **Failure detection and correlation:** When **the High Availability Manager** receives a report that a host appears to be *down*, it cross-checks the host’s status across reports from other agents to avoid false positives while ensuring a rapid response to true outages. Once a host is confirmed *down* via peer agent reports, a **150‑second cooldown** timer starts:
* If the host recovers before the timer expires, the event is cleared, and no action is taken. This is to prevent routine reboots from triggering VM evacuations.
* If the host remains *down* when the timer ends, **High Availability Manager** confirms the failure and emits a *host-down* notification to the **VM Evacuation Service**.
6. **Automated VM evacuation:** Upon receiving the *host‑down* notification, the **VM Evacuation Service** verifies the host’s state, collects all resident VMs, and evacuates them **one VM at a time** to a healthy host in the cluster. Target selection honors existing placement constraints (such as aggregates or affinity rules).
7. **Continuous protection loop:** After evacuation is complete, normal monitoring resumes. Host Agents continue probing both the recovered host (if it comes back online) and all remaining hosts, enabling VM HA to maintain an always-on protection loop with no manual intervention. Note that if a host that went offline comes back online, VM HA will not automatically migrate the VMs that were originally located on this host back to it. The Dynamic Resource Rebalancing (DRR) service will make VM-balancing decisions independently when resource contention occurs on any host.
## UI Observability
### VM HA Status Across Clusters
The PCD UI shows the current status of VM HA for each cluster at the following locations:
* The PCD home page displays a Cluster VM HA Status widget.
* The Infrastructure > Clusters page. Hovering over the VMHA status in the table displays the status for each prerequisite.
* The VM High Availability pane on the Cluster details page. Expanding the card will show the status for each individual prerequisite.
The possible VM HA statuses per cluster are:
* **Protected**: Shows the number of clusters with VM HA enabled that meet all prerequisites.
* **Degraded**: Shows the number of clusters with VM HA enabled that have a single point of failure (for example, only one host in the cluster has the persistent storage or image library role). Such clusters can absorb host failures as long as the host is not the single point of failure.
* **Not Protected**: Shows the number of clusters with VM HA enabled that have not met one or more prerequisites.
* **Disabled**: Shows the number of clusters for which VM HA is disabled.
### VM HA Events
When an active VM HA event is in progress, a banner message is displayed in the PCD UI with details about the event. The banner message will persist until evacuations are complete or the banner is manually dismissed.
The Host details page for a host that experienced an outage shows the evacuation status for all VMs on the host. The details will be displayed for 24 hours after the host outage. The VMHA Past Events table on the Host details page will show all historical VM HA events that occurred on the host. Clicking View Details displays detailed VM evacuation status for the selected event. From the detailed VM evacuation status page, you can retry failed evacuations (for all failed evacuations or for individual VMs) for the most recent VM HA event.
Note that VMs in error, unknown, rescued, or resized status (i.e., the VM has been resized but the resize has not been confirmed) will not be evacuated.
## VM HA Interoperation with Other Services
This section describes how VM HA interoperates with other services configured for your cluster.
### Host Aggregates
VM HA will honor [Host Aggregates](/private-cloud-director/virtualized-clusters/host-aggregate) and migrate VMs to another host from the same host aggregate.
Here are some limitations you might need to consider:
* VM HA requires that all hosts in a cluster belong to the same host aggregate or have no aggregate assignment. Mixed aggregate configurations will prevent VM HA from being enabled.
* All hosts from a host aggregate must belong to a single cluster and not span multiple clusters.
* If a host aggregate has only one host that goes down, VM HA cannot find a suitable migration target, causing VMs to enter an error state.
### DRR
DRR and Virtual Machine High Availability are designed to interoperate well together. A host failure event may occur while DRR is actively rebalancing VMs, either from the same host or from other hosts in the cluster. When this happens:
1. VM HA will detect the host failure and initiate VM evacuations
2. The VM evacuations may result in cluster imbalance
3. DRR will then detect the imbalance during either the current run or the next run.
4. DRR will redistribute the load across the cluster to address the imbalance.
### VMs with Hard Affinity or Anti-Affinity Rules
* **Hard affinity**: VM HA will temporarily break hard affinity to evacuate VMs with hard affinity. The evacuation process will attempt to locate a suitable target host to accommodate all VMs belonging to the hard affinity server group.
* **Soft affinity**: VM HA identifies a host with sufficient capacity to host all VMs in the affinity group and migrates all VMs sequentially to the target host. If VM HA cannot find a suitable target host, VM HA will migrate VMs to available hosts, potentially violating the soft affinity policy.
* **Hard anti-affinity**: VM HA will fail to evacuate VMs with hard anti-affinity, and VMs will land in `Error` state. This will be addressed in an upcoming release.
* **Soft anti-affinity**: VM HA will attempt to find target hosts that satisfy the anti-affinity policy for each VM to be migrated, ensuring that all VMs part of the soft anti-affinity policy are placed on separate hosts. If VM HA cannot find a suitable target host, VM HA will evacuate the VMs to available hosts, potentially violating the soft anti-affinity policy.
{% hint style="info" %}
**NOTE**
If VM evacuation fails in any of the scenarios listed above, the VM may still appear as active in the UI.
{% endhint %}
### VM States
VMs in a suspended or paused state will not be evacuated to another host.
### VMs with Special Properties
1. **Virtual TPM-enabled VMs:** VM HA will evacuate VMs with [Virtual TPM](/private-cloud-director/virtualized-clusters/virtual-tpm) enabled, provided the [vTPM HA prerequisites](#vm-ha-pre-requisites) are met.
**Resized VMs:** VM HA will skip evacuating a VM that has been resized but not confirmed yet.
## Supported Scale
VM HA is currently supported for up to 400 hosts per region.
## Known Issues and Limitations
* VM HA is presently not supported for clusters with GPUs enabled. This support will be added in a future release of Private Cloud Director
* VM HA relies on connectivity with the management plane. If network connectivity to the management plane is degraded, VM HA performance will be negatively affected. If the management plane experiences an outage, VM HA will not be operational for the duration of the outage.
## Troubleshooting VM HA
For step-by-step diagnostics when VM HA does not behave as expected, see the VM HA troubleshooting runbook:
[Troubleshoot VM HA](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/troubleshoot-vm-ha)
The runbook covers:
* A host failed but VMs were not evacuated — how to verify cluster settings, host detection, HA agent health, and libvirt exporter liveness.
* Consul health prerequisites — how to check and recover the distributed coordination service that VM HA depends on for failure detection.
* Shared and FC storage validation — how to confirm that storage is reachable on all hosts before relying on VM HA.
* Errors when enabling or disabling VM HA — prerequisites to verify and how to recover from 404 or 503 responses.
* Post-upgrade re-validation — how to confirm VM HA is functional after a host or management-plane upgrade.
# Host
Learn how to add and configure Hosts in Private Cloud Director for your virtualized cluster. Discover Host roles, management, and troubleshooting tips to ensure optimal performance and uptime for your
A Host is a physical machine that you supply to space.vars.product\_name as a hypervisor. Each Host contains the resources needed for your cluster, such as virtual machines, storage, and networking components. Once authorized and configured, you can deploy virtual machines on top of the Host.
You can add multiple Hosts to your space.vars.product\_name virtualized cluster. After you configure your Cluster Blueprint, space.vars.product\_name has the information it needs to configure Hosts that you add to the cluster.
Learn how you can add a Host to your virtualized cluster, assign roles, and configure it for production use. You will also learn how to manage the Host lifecycle and troubleshoot common issues.
## Hypervisor
space.vars.product\_name standardizes on and uses open source KVM hypervisor behind the scenes. KVM is a type 1 hypervisor that runs directly within the Linux kernel on the host hardware. KVM hypervisor relies on and leverages hardware virtualization extensions from x86 and/or AMD processors to provide full virtualization capabilities.
## Resource Management
Hypervisor resource management enables you to make the best use of CPU and Memory resources available across all your hypervisor nodes within a virtualized cluster, to ensure that:
* Your applications get the required resources when they need, to meet your business SLA.
* You make the most optimal utilization of available resources within your cluster, and prevent or reduce and waste.
Resource management in space.vars.product\_name has multiple components to it. At the Hypervisor level, resource management is handled via a combination of the following components.
### Memory Ballooning
Memory ballooning is a KVM hypervisor technique for dynamic memory management that is used to reduce the impact of memory-overcommitment on the hypervisor load. Memory ballooning allows your guest OS to dynamically evict unused pages of virtual memory, so that the KVM host can then share the unused memory with other VMs, allowing the host to overcommit memory and optimize resource use by giving memory to VMs that suddenly need it. space.vars.product\_name hypervisor makes use of a combination of memory ballooning and memory swapping to enable optimal utilization of host memory.
### Memory Deduplication & Consolidation
Memory deduplication and consolidation is a memory management feature used by KVM hypervisor to consolidate identical memory pages to achieve higher virtual machine density. KVM hypervisor achieves this using a linux feature called **Kernel Samepage Merging (KSM)**, that finds identical memory pages across different virtual machines and merges them into a single, shared, Copy-on-Write (CoW) page, significantly reducing overall memory usage. KSM enables space.vars.product\_name to improve VM memory density on a host without impacting performance.
### Resource Overcommitment / Allocation Ratios
### Memory Overcommitment (Allocation Ratio)
space.vars.product\_name allow you to overcommit memory (also called memory overprovisioning or memory allocation ratio) at each host level, enabling you to effectively share the host level memory resources across virtual machines. Each new hypervisor host provisioned in space.vars.product\_name is configured with a default memory **overcommitment ratio of 1:1.5**. This means by default each host's memory is overcommitted by 50%. You can change this configuration by navigating to the 'Cluster Hosts' view in the space.vars.product\_name UI, then selecting a specific host, and editing the allocation ratios.
The KVM hypervisor uses a combination of memory ballooning and memory swapping to manage memory across running virtual machines.
### CPU Overcommitment & Time-Sharing (Allocation Ratio)
space.vars.product\_name allows you to overcommit CPU resources (also called CPU overprovisioning or CPU allocation ratio) at each host level, enabling you to effectively share the host's CPU cores across running virtual machines.
Behind the scenes, the KVM hypervisor dynamically allocates physical CPU cores to running virtual machines, context switching between different virtual machines, to enable sharing and overcommit.
Each new hypervisor host provisioned in space.vars.product\_name is configured with a default CPU **overcommitment ratio of 1:16**. This means by default each host's CPU is overcommitted by 16x. We choose this default setting as most virtualized workloads tend to be memory bound and can share CPU without impacting workload performance. But when running CPU heavily workloads, you will need to change this default overcommitment ratio to suite your application's needs. You can also utilize features like CPU pinning to reduce the context switching overhead for CPU heavy workloads.
### Ephemeral Storage Overcommitment (Allocation Ratio)
space.vars.product\_name also allows you to overcommit the [Ephemeral Storage](/private-cloud-director/storage/ephemeral-storage#what-is-ephemeral-storage) at per host level (also called ephemeral disk overprovisioning or ephemeral disk allocation ratio). This capability makes it possible for you to over-allocate the disk storage at per hypervisor host level that is used to store root disks for VMs using ephemeral storage for their root disks. Useful when you do not expect all VMs using ephemeral storage to use all their disk space.
For example, lets say that your VM ephemeral storage path on your hypervisor is set to the default location of `/opt/data/instances` and this path is on your root partition which has 100GB of total disk space. If you set the ephemeral disk overcommitment value to 2, this means you can create virtual machines with the sum total of root disk size not exceeding 200GB. This can become a problem if all the VMs start fully utilizing their allocated root disk space.
Each new hypervisor host provisioned in space.vars.product\_name is configured with a default ephemeral storage **overcommitment ratio of 1:9999**. This means by default each host's ephemeral disk is overcommitted by 9999x. We set it to such a high value by default because most production environments use block storage volumes for production VM root disks and VMs that use ephemeral storage for their root disk often tend to not utilize it fully. You will need to update this default value if you plan to use and rely on ephemeral root disk storage for production virtual machines.
{% hint style="warning" %}
Each new hypervisor host is configured with a default ephemeral storage **overcommitment ratio of 1:9999**.
You must **update this default** if you plan to use and rely on ephemeral root disk storage for production virtual machines. We recommend updating it to 1 for no overcommitment, or to a reasonable overcommitment ratio based on the nature of your VM workloads and their expected use of root disk.
{% endhint %}
### View Host CPU, Memory and Storage Allocation and Utilization Info
The current allocation and utilization values for CPU, Memory and Epheremeral Strorage for each host are reported as individual columns as part of the Cluster Hosts grid view in the UI.
* For Compute:
* **% CPU utilized** - this is the current CPU utilization for this host. This value is an aggregate across all physical CPU cores on the host and is reported in terms of total vs available MHz.
* **% VCPUs allocated** - this represents the sum total of VCPUs that have been allocated across all virtual machines provisioned on this host. For eg, if the host has 5 VMs currently deployed on it with 4 CPUs each, you will see the VCPUs allocated value to be 20. If the host has 10 physical CPUs and has overcommitment ratio of 1:2, then you can allocate a maximum of 20 VCPUs on this host.
* For Memory:
* **% Memory utilized** - this is the current Memory utilization for this host, reported as total vs available GiB.
* **% Memory allocated** - this represents the sum total of RAM that have been allocated across all virtual machines provisioned on this host. For eg, if the host has 5 VMs currently deployed on it with 4 GB RAM each, you will see the Memory allocated value to be 20. If the host has 20 GB physical RAM and has overcommitment ratio of 1:1, then you can allocate a maximum of 20 GB of memory across all VMs on this host.
* For Storage:
* **% root disk used** - this is the current root disk utilization for this host, reported as total vs available GiB.
* **% ephemeral disk used** - this represents the sum total of ephemeral root disks that have been allocated across all virtual machines provisioned on this host that are using ephemeral root disks. For eg, if the host has 5 VMs currently deployed on it, each using 10 GB of ephemeral root disk, you will see the ephemeral disk used value to be 50GB. If the host has 250 GB physical disk and has ephemeral disk overcommitment ratio of 1:1, then you can allocate a maximum of 250 GB of ephemeral root disk across all VMs on this host.
### View and Edit Overcommitment / Allocation Ratios
You can edit CPU, memory or ephemeral disk overcommitment values (allocation ratios) at per host level by following these steps:
1. Navigate to the Cluster Hosts view in the UI,
2. Select a specific host, then click on the host name to go to the host details view. Here you will see the currently set values for allocation ratios for CPU, Memory and Ephemeral storage for this host under "Properties" section.
3. To edit the current values, go back to the Host grid view, select the host then choosing "Edit allocation ratio" as the action from the action bar.
4. Here you can see again the currently configured overcommitment values for CPU, Memory and ephemeral disk for this host.
5. You can reset these values to their system default or change specific values here.
## Understanding Host Agent and Roles
### Host Agent
The Platform9 Host agent is the first component you install for each Host. The Host agent enables you to add Hosts and configure their roles in your virtualized cluster. Based on the assigned role to each Host, the agent downloads and configures the required software, integrating with the space.vars.product\_name management plane.
The Host agent also provides ongoing health monitoring of the Host, including the detection of failures and errors. It helps Platform9 orchestrate upgrades when you choose to upgrade your space.vars.product\_name deployment to a newer version.
### Host Roles
As part of the Host authorization process, you can configure Hosts to perform specific functions by assigning them one or more roles. The following roles are supported:
**Hypervisor**
The Hypervisor role enables the Host to function as a KVM-based hypervisor in your virtualized cluster. It is recommended you assign this role to all Hosts in your cluster, unless you experience performance bottlenecks and want to avoid running VM workloads on select Hosts that have other roles such as image library or storage roles.
**Image Library**
Every cluster needs at least one Image Library, which hosts the cluster copy of virtual machine source images from which you can provision new VMs. See more on [Configuring Image Library Role here](/private-cloud-director/images-and-image-library/image-library---images#configuration-).
**Persistent (Block) Storage**
You can typically configure one or more Hosts in your cluster as a **Block Storage Node**. See more on [Block Storage Service Configuration](/private-cloud-director/storage/block-storage#block-storage-service-configuration).
**Advanced Remote Support**
For specific troubleshooting situations, Platform9 support teams may request access to gather detailed telemetry from a Host that experiences problems automatically. This mode is turned off by default. To enable Advanced Remote Support, contact Platform9 Support.
**DNS**
Enables DNS as a Service (DNSaaS), which is an optional component.
## Prerequisites
Before adding a Host, ensure that your Host meets the prerequisites and that you have configured in your Cluster Blueprint.
* Verify that your Host meets the [Pre Requisites](/private-cloud-director/getting-started/pre-requisites) for space.vars.product\_name.
* Ensure you have administrative access to the Host.
* Confirm that your [Virtualized Cluster Blueprint](/private-cloud-director/virtualized-clusters/virtualized-cluster-blueprint) is configured in the space.vars.product\_name console.
* Confirm that you have created a Cluster in the space.vars.product\_name console.
## Add a Host
To add a host to space.vars.product\_name, you need to follow the steps to install the space.vars.product\_name agent software on your physical host, then follow the steps to add it to space.vars.product\_name.
#### Step 1: Add a Host
The process for adding a host varies depending on which Platform9 product you are using. Choose the appropriate method below based on your deployment type:
* SaaS Deployment
* Self-Hosted Deployment
* Community Edition Deployment
{% tabs %}
{% tab title="SaaS Deployment" %}
For SaaS deployments, adding a Host is on the space.vars.product\_name console with minimal configuration requirements.
1. Navigate to **Infrastructure > Cluster Hosts** on space.vars.product\_name console.
2. Select **Add New Hosts**
3. Follow the on-screen instructions. You are required to run these [pcdctl](/private-cloud-director/reference/pcdctl-command-line) commands using `sudo` privileges. The command requires you to add values specific to your environment that are provided on your space.vars.product\_name console. It takes about 2-3 minutes to download and install the Platform9 Host agent and other necessary Platform9 software components.
You have now successfully added a Host to your SaaS deployment. Continue to **Step 2: Authorize Host and Assign Roles**
{% endtab %}
{% tab title="Self-Hosted Deployment" %}
For self-hosted deployments, you may need to manually configure DNS entries before adding the Host.
**Step 1: Add a Host to Self-Hosted Deployment**
{% hint style="info" %}
**NOTE**
Hypervisor Hosts deployed as virtual machines must have virtualization support available inside the VM. Virtual machines on ARM CPUs are currently untested.
{% endhint %}
1. Verify nested virtualization
You can choose to verify how the nested virtualization works in a VM. Check for virtualization support inside the VM by running:
{% tabs %}
{% tab title="Bash" %}
```bash
egrep "svm|vmx" /proc/cpuinfo
```
{% endtab %}
{% endtabs %}
2. Add DNS entries to each Host
An FQDN is a fully-qualified domain name for your space.vars.product\_name installation. You will need both infrastructure and workload region FQDNs for your self-hosted deployment.
As a root user, add DNS entries on the hypervisor Host for both the infrastructure and workload region FQDNs: `/etc/hosts` file:
{% tabs %}
{% tab title="Bash" %}
```bash
echo "" | tee -a /etc/hosts
echo "" | tee -a /etc/hosts
```
{% endtab %}
{% endtabs %}
Replace `` with your management plane IP address and the FQDNs with your specific domain names.
Here is a sample example:
{% tabs %}
{% tab title="Bash" %}
```bash
echo "10.9.11.246 pcd.pf9.io" | tee -a /etc/hosts
echo "10.9.11.246 pcd-community.pf9.io" | tee -a /etc/hosts
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**Important!**
Before adding a hypervisor Host, ensure that you have saved a Cluster Blueprint and created a Cluster on the space.vars.product\_name console.
{% endhint %}
* Navigate to **Infrastructure > Cluster Hosts** and the select **Add New Hosts**.
* Follow the on screen instructions. You are required to enter the administrative user password when prompted. For more details on `pcdctl` CLI see [Pcdctl Command Line](/private-cloud-director/reference/pcdctl-command-line)
You have now successfully prepared your Host for self-hosted deployment. Continue to **Step 2: Authorize Host and Assign Roles**
{% endtab %}
{% tab title="Community Edition Deployment" %}
For Community Edition deployments, the process is similar to self-hosted but uses a specific FQDN for the user interface, unless configured otherwise via a Custom Install.
**Step 1: Add a Host to Community Edition Deployment**
{% hint style="info" %}
**NOTE**
Hypervisor Hosts deployed as virtual machines must have virtualization support available inside the VM. Virtual machines on ARM CPUs are currently untested.
{% endhint %}
1. **Verify nested virtualization (for VM Hosts)**
If you want to verify that nested virtualization works in a VM, check for virtualization support inside the VM:
{% tabs %}
{% tab title="Bash" %}
```bash
egrep "svm|vmx" /proc/cpuinfo
```
{% endtab %}
{% endtabs %}
If the command returns results, your VM supports nested virtualization and can run other virtual machines.
2. **Configure DNS entries**
By default, Community Edition uses `pcd.pf9.io` for the user interface and to reference the workload region, unless you have customized it [Custom Installation](/private-cloud-director/getting-started/getting-started-with-community-edition/custom-installation).
* Workload region & UI FQDN: `pcd.pf9.io`
3. **Add DNS entries to your Host**
Log in to your Host as a root user and add DNS entries for the FQDN by running the following command:
{% tabs %}
{% tab title="Bash" %}
```bash
echo "" | tee -a /etc/hosts
```
{% endtab %}
{% endtabs %}
Replace `` with your management plane IP address and the `FQDN` with your specific domain name.
Here is a sample example:
{% tabs %}
{% tab title="Bash" %}
```bash
echo "10.9.11.246 pcd.pf9.io" | tee -a /etc/hosts
```
{% endtab %}
{% endtabs %}
4. **Add the Host through the** space.vars.product\_name **console.**
{% hint style="info" %}
**Important!**
Before proceeding, ensure that you have saved a Cluster Blueprint and Created a cluster in the space.vars.product\_name console.
{% endhint %}
* Navigate to **Infrastructure > Cluster Hosts** and then select **Add New Hosts**.
* Follow the on screen instructions. You are required to enter the administrative user password when prompted. For more details on `pcdctl` CLI see [Pcdctl Command Line](/private-cloud-director/reference/pcdctl-command-line)
You have now successfully prepared your Host for Community Edition deployment. Continue to **Step 2: Authorize Host and Assign Roles**
{% endtab %}
{% endtabs %}
#### Step 2: Authorize Host and Assign Roles
A successfully configured Host is accessible on **Infrastructure > Cluster Hosts** with an `Unauthorized` status, indicating an authorization and cluster role assignment.
1. Navigate to **Infrastructure > Cluster Hosts** and select a specific Host.
2. Select **Edit Roles** to configure the appropriate roles for the Host based on your cluster architecture.
3. Configure roles based on your requirements:
* **Hypervisor Role**: Enables the Host to function as a KVM-based hypervisor. Read more on [Hypervisor Role](#hypervisor-role).
* **Networking Service Configuration:** Select appropriate Host Network Config for the Host networking requirements. Read more on [Networking Service Configuration](/private-cloud-director/virtualized-networking/networking-overview#networking-service-configuration).
* **Image Library Role:** Configures the Host to store VM images for the cluster. Read more on [Configuring Image Library Role here](/private-cloud-director/images-and-image-library/image-library---images#configuration-).
* **Block Storage Role:** Enables the Host to provide persistent storage services. Read more on [Configure a Host with Block (Persistent) Storage Node Role](/private-cloud-director/storage/volume#configure-a-host-with-block-persistent-storage-node-role).
* **Advanced Remote Support:** Enables Platform9 support to gather detailed telemetry for troubleshooting purposes. Read more on [Enabling Advanced Remote Support](#enable-advanced-remote-support).
* **DNS Checkbox**: Enables DNS as a Service (DNSaaS), which is an optional component. The DNS checkbox is for [Dns As A Service Dnsaas ](/private-cloud-director/virtualized-networking/dns-as-a-service-dnsaas). Read more on [Configuring DNS-as-a-Service](/private-cloud-director/virtualized-networking/dns-as-a-service-dnsaas#configuration).
4. Select **Update Role Assignment**
The space.vars.product\_name management plane works with the Platform9 agent installed on your Host to configure the required software. This process typically takes 3-5 minutes to complete. During this time, your Host status changes to `converging` in the space.vars.product\_name console.
You have successfully authorized your Host and assigned roles. Your Host is now being configured for use in your cluster.
#### Step 3: Monitor Host addition status
While your Host is in the `converging` state, you can monitor the configuration progress by examining the Host agent log files.
1. Locate the Host agent log file.
The Host agent log files are located on your Host. See the [Log Files](https://docs.platform9.com/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files#log-files) section for detailed information about log file locations. The primary Host agent log is located at `/var/log/pf9/hostagent.log` on your Host.
2. Monitor the configuration progress.
Tail the log file to monitor the status of host addition:
{% tabs %}
{% tab title="Bash" %}
```bash
tail hostagent.log
```
{% endtab %}
{% endtabs %}
3. Verify successful completion
Monitor the log output for completion indicators. Once the Host is authorized and the role assignments have taken effect, your Host status changes from `converging` to `ok` in the space.vars.product\_name console.
Your Host is now ready to use and can run workloads according to its assigned roles.
## Manage Host lifecycle
### Remove a Host
Entirely removing a Host from your space.vars.product\_name setup is a two-step process. You must first remove all roles assigned to the Host, and then, if necessary, decommission the Host to clean up any space.vars.product\_name related data associated with it. You must perform both steps if you plan to re-add the Host to your current or another product name setup.
#### Step 1 - Remove all roles and deauthorize a Host
Removing all roles from a Host is the first step toward entirely removing a Host from your space.vars.product\_name setup. Removing all roles uninstalls any specific packages and software components assigned to the Host.
**Prerequisites before removing a Host from** space.vars.product\_name setup.
* If the host is assigned `hypervisor` role, make sure that no VMs are running on the host using `sudo virsh list --all` The expected output is an empty list of VM UUIDs and their corresponding statuses.
* If the host is assigned a `persistent storage` role, make sure that this host is serving no storage volumes.
* If the host is assigned `image library` role, ensure that the host is serving no images in the image library. On the image library host, check if the image UUID exists in the `/var/lib/glance/images/glance` default directory, or if a custom image directory was added, check that location.
1. Navigate to the **Cluster Hosts** on the console.
2. Select the specific Host.
3. From the Actions bar dropdown, select **Remove all roles**.
**Remove all roles using CLI:**
Use the following `pcdctl` command to remove all roles from a Host:
{% tabs %}
{% tab title="Bash" %}
```bash
pcdctl deauthorize-node
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**NOTE**
Removing all roles only uninstalls and removes software packages associated with all roles from the Host. It does not clean up and delete any directories or files created as part of the installation of these software components. You must decommission a Host for all space.vars.product\_name related data from that Host to be removed.
{% endhint %}
#### Step 2 - Decommission a Host
When you remove all roles from a Host using the command above, any space.vars.product\_name specific packages and software components associated with those roles are uninstalled and removed from the Host. However, any directories where the packages were installed are not cleaned up or deleted. This ensures that you still have access to the log files for those components if required for debugging.
To remove these directories and clean up any space.vars.product\_name related data from the Host, you need to run the decommission Host command.
**Prerequisites:**
* You must remove all roles from the Host using the space.vars.product\_name console or CLI before decommissioning.
* Always back up important data, such as log files and configuration files, from the Host before decommissioning.
* You must decommission a Host before you can add it again to your current or any other space.vars.product\_name setup. Not doing so results in problems when re-authorizing the Host in the space.vars.product\_name setup.
**Decommission using CLI:**
Currently, decommissioning a Host can only be performed using `pcdctl` CLI.
Use `pcdctl` to decommission a Host by running the following command:
{% tabs %}
{% tab title="Bash" %}
```bash
$ pcdctl decommission-node
Do you wish to decommission the node? (y/n) y
Checking if any roles exist on the host
Cleaning up the node
Decommission Successful
```
{% endtab %}
{% endtabs %}
Once the command executes successfully, the Host is removed from the list of active Hosts in the space.vars.product\_name console. If you encounter errors during decommissioning, check the logs for details and ensure the Host is reachable.
## Host Properties
### Host ID
Each hypervisor host gets a system-assigned ID when it's created. By default, the ID value is not shown in the host grid UI, but you can view the ID information by clicking on the 'Manage Columns' button on the cluster hosts grid view, then selecting the ID field to be displayed. You can also view a host's ID on the host details view by clicking on the host name from the host grid view. You can also query it from the `pcdctl` CLI by running `pcdctl hypervisor list` command or `pcdctl hypervisor show ` command where `` is the name of your hypervisor host.
### Host Connection Status
Host connection status, represented by the 'Connection Status' column in the Cluster Hosts grid in the UI, represents the status of connectivity between the host agent and the space.vars.product\_name management plane. The following are the different status values:
* **online** - The Host agent is connected to the space.vars.product\_name management plane and responds to heartbeats.
* **offline** - The Host agent is unable to connect to the space.vars.product\_name management plane due to the host being in a powered-off state or because the Host is running. However, the Platform9 host agent may be experiencing issues connecting with the management plane. For more information on debugging steps, refer to [Host Issues](https://platform9.com/docs/private-cloud-director/troubleshooting/hypervisor-or-host-issues).
### Host Role Status
Host role status, represented by the 'Role Status' column in the Cluster Hosts grid in the UI, indicates whether the host has been added to any virtualized cluster and assigned roles within the cluster. The following are the different role status values:
* **unauthorized -** The host has space.vars.product\_name host agent installed but has not yet been added to a cluster or assigned any specific roles.
* **applied** - The host is assigned to a cluster, has specific roles assigned to it within that cluster, and those roles have been successfully applied.
* **converging** - A new role is being applied to the host and/or the host is being authorized and added to a cluster
* **failed / error -** The host appears to be in a failed or error state. For more information, refer to [Host Issues](https://platform9.com/docs/private-cloud-director/troubleshooting/hypervisor-or-host-issues).
* **unknown** - The role status is displayed as unknown when the host connection status is offline, as it does not know if the application of any roles was in progress or if it was successful.
### Host OverCommitment / Allocation Ratios
You can update the overcommitment / allocation ratios for individual hosts by navigating to the Cluster Hosts grid view in the UI, slelecting a specific host, then selecting "Update allocation ratios" from the actions menu.
Read more about [#memory-overcommitment-allocation-ratio](#memory-overcommitment-allocation-ratio "mention") [#cpu-overcommitment-and-time-sharing-allocation-ratio](#cpu-overcommitment-and-time-sharing-allocation-ratio "mention") and [#ephemeral-storage-overcommitment-allocation-ratio](#ephemeral-storage-overcommitment-allocation-ratio "mention").
### Storage Initiator Identifiers (IQN and WWN)
Each host exposes the storage initiator identifiers that backends use to authenticate connections. Both appear as columns on the **Cluster Hosts** grid under **Infrastructure > Cluster Hosts**.
* **IQN** (iSCSI Qualified Name) – The host's iSCSI initiator name. iSCSI storage backends use this to recognize and authorize the host. Format: `iqn.YYYY-MM.:`.
* **WWN** (World Wide Name) – The host's Fibre Channel initiator port identifier. FC backends use this for zoning and host access configuration on the array side.
Use these when configuring host access on the storage array or when correlating a volume's **Management Host** (see [Volume](/private-cloud-director/storage/volume#placement-information)) with the array's view of attached hosts.
## View Volumes Mounted on a Host
Each host's details page includes a **Volumes** tab that lists the volumes currently mounted on virtual machines running on that hypervisor. Navigate to **Infrastructure > Cluster Hosts**, select a host, then open the **Volumes** tab.
The **Mounted Volumes** table shows, per volume: **Name**, **Status**, **Management Host** (the block storage host serving the volume), **Volume Backend**, **Type**, **VM** (the virtual machine the volume is attached to, linked to the VM details page), **Hypervisor Host** (matches the host you're viewing), **Device** (the guest device path, for example `/dev/vda`), **Snapshots**, **Size**, and **Bootable**.
Use this view to scope a storage problem to one hypervisor: which volumes is it currently serving, which VMs depend on them, and which backends are in play. Toggle **View Volumes in All Tenants** to widen the scope beyond the current tenant.
## Host Aggregates
A host aggregate is a **group of hosts** within your virtualized cluster that share common characteristics. Read [Host Aggregate](/private-cloud-director/virtualized-clusters/host-aggregate) for more information on how to configure them.
## Debugging Compute Service Problems
Follow [Troubleshooting And Log Files](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files) for steps to troubleshoot issues with the compute service.
# Maintenance Mode
space.vars.product\_name maintenance mode enables administrators to perform routine maintenance operations on their fleet of hosts in the data center in a safe and orchestrated manner. Maintenance mode allows you to safely migrate virtual machines from a host and prevent new virtual machines from being scheduled during maintenance operations.
## What Happens in Maintenance Mode?
When maintenance mode is enabled for a host:
* All running VMs on the host are automatically live-migrated to another compute host in the cluster.
* You have the option to choose the host to which the VMs will get migrated
* The host is marked as unschedulable, ensuring no new VMs are placed on it.
* Once all VM migrations are complete, the host remains online but isolated from scheduling operations, allowing administrators to perform software updates, hardware maintenance, or diagnostics.
## Enabling Maintenance Mode
To enable maintenance mode on a host:
1. Navigate to Infrastructure → Cluster Hosts in the space.vars.product\_name UI.
2. Select the target host from the list (optional). If no target host is selected, maintenance mode will select an appropriate target host from the cluster.
3. Click Other → Enable Maintenance Mode.
As noted in point 2 above, you may choose to:
* Allow space.vars.product\_name to automatically determine the best destination for each VM based on cluster capacity and placement policies.
* Manually specify a destination host for all VM migrations.
Click Migrate VMs and Enable Maintenance Mode to initiate the process.
After host maintenance operations are completed, you can disable maintenance mode to allow the host to rejoin the scheduler.
## Disabling Maintenance Mode
Before disabling maintenance mode, ensure that any maintenance tasks on the host have been completed successfully and that the host is healthy and ready to handle new workloads.
1. Navigate to Infrastructure → Cluster Hosts.
2. Locate the host that is currently in Maintenance Mode.
1. Hosts in maintenance mode are typically marked with a "Maintenance Mode" label or status indicator.
3. Select the host, then click on Other → Disable Maintenance Mode
The host will be marked as schedulable again and space.vars.product\_name will resume placing new virtual machines on this host.
## UI Observability
The Private Cloud Director UI provides real-time status indicators and detailed migration progress information throughout the maintenance mode lifecycle.
### Maintenance Mode Status on the Cluster Hosts Page
The **Infrastructure → Cluster Hosts** page reflects the current maintenance mode state for each host:
* **Scheduling column**: When maintenance mode is enabled on a host, the Scheduling column displays **Disabled** to indicate that the host is no longer accepting new VM placements.
* **Maintenance Mode label**: During the transition into maintenance mode, a status label displays **Entering Maintenance Mode** in the Scheduling column. This label is visible in the hosts table for any host that is currently undergoing the maintenance mode process.
* **Other Actions menu**: When a host is in maintenance mode, the **Enable Maintenance Mode** option in the Other Actions dropdown is disabled, and the **Disable Maintenance Mode** option becomes available. The **Enable/Disable Scheduling** option is also disabled while maintenance mode is active, since scheduling is managed by the maintenance mode process.
### Maintenance Mode Status on the Host Details Page
Clicking on a host that is in maintenance mode opens the Host details page, which displays a banner at the top of the page with the following information:
* **Migration status summary**: The banner indicates whether VM migrations are in progress or have completed successfully.
* **VM count**: The total number of VMs that were migrated out of the total that will be migrated.
* **Timestamps**: The start time of the maintenance mode operation is displayed. The completion time is displayed once all migrations are finished.
* **See Details**: A button that opens the View Migration Progress panel with detailed per-VM migration status.
* **Disable Maintenance Mode**: A button that allows you to disable maintenance mode directly from the Host details page without navigating back to the hosts list.
### View Migration Progress
Clicking **See Details** on the Host details page banner opens the View Migration Progress panel. This panel provides a detailed breakdown of the maintenance mode operation:
* **Overall Migration Status**: Displays the current state of the migration operation (for example, Completed, In Progress, or Failed).
* **Number of VMs Migrated**: Shows the count of VMs that have been successfully migrated out of the total.
* **Start Time**: The timestamp when the maintenance mode operation was initiated.
* **Completion Time**: The timestamp when all migrations finished (displayed upon completion).
* **Virtual Machines Migration table**: Lists each individual VM that was migrated, along with its migration status. This allows administrators to identify any VMs that may have failed to migrate and retry migration.
## Maintenance Mode Interoperation with Other Services
This section describes how maintenance mode interoperates with other services configured for your cluster.
### Host Aggregates
Maintenance mode will honor[ Host Aggregates](/private-cloud-director/virtualized-clusters/host-aggregate) and migrate VMs to another host from the same host aggregate.
Limitations:
* All hosts from the Host Aggregate must belong to a single cluster and not span multiple clusters.
* If the host aggregate has a single host that has gone down, maintenance mode will not find a suitable target to migrate the VMs, and maintenance mode will fail.
### DRR
A host in maintenance mode will have scheduling VMs disabled while maintenance mode is active. Consequently, DRR will exclude any hosts that are in maintenance mode in a given cluster, and only balance workloads on the remaining active hosts.
### VMs with Hard Affinity or Anti-Affinity Rules
* **Hard affinity**: Maintenance mode will fail if one or more VMs with hard affinity rules are in a powered-on state.
* **Soft affinity**: Maintenance mode will attempt to keep all VMs in the affinity group together by migrating the first VM to a randomly selected host in the cluster and then migrating the subsequent VMs to the same host until it reaches capacity.
* **Hard anti-affinity**: Maintenance mode will honor the anti-affinity policy by placing the VMs in the anti-affinity group on separate hosts. If no appropriate target host is found to migrate a VM, maintenance mode will fail.
* **Soft anti-affinity**: Maintenance mode will attempt to find target hosts that satisfy the anti-affinity policy for each VM to be migrated in a powered-on state, ensuring that all VMs part of the soft anti-affinity policy are placed on separate hosts. If maintenance mode cannot find a suitable target host, maintenance mode will migrate the VM(s) to a target host that violates the soft affinity policy, and the policy violation will be displayed on the UI.
### VM States
Maintenance mode will migrate VMs that are in an active state (powered on).
Maintenance mode will not migrate VMs with the following statuses:
* Shutdown (powered off)
* Suspended
* Error
* Unknown
Maintenance mode will not proceed if a VM exists in the following states:
* **Pending resize confirmation**: The resize operation needs to be confirmed for the VM to be eligible for migration by maintenance mode.
* **Rescued**: The VM must be un-rescued for the VM to be eligible for migration by maintenance mode.
### VMs with Special Properties
1. **Virtual TPM-enabled VMs:** Maintenance mode does not have the ability to live migrate VMs with[ Virtual TPM](/private-cloud-director/virtualized-clusters/virtual-tpm) enabled today. This ability is coming soon.
2. **Zero flavor VMs:** Maintenance mode will migrate VMs created with zero flavor in a powered-on state.
3. **VMs with hot-added CPU or memory:** Maintenance mode will migrate VMs that have hot-added CPU or memory resources.
## Troubleshooting
If VMs fail to migrate, or are left stranded or in an error state during maintenance mode, see [Troubleshoot Maintenance Mode Migration Failures](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/troubleshoot-maintenance-mode-migrations).
# CPU Model
## What is a CPU Model
In space.vars.product\_name, a CPU model refers to the set of CPU features and flags exposed to a virtual machine. This is how space.vars.product\_name compute service manages the CPU architecture presented to a virtual machine guest operating system, allowing for flexibility and control over performance, compatibility, and migration capabilities.
## Use Cases for CPU Model
* **Performance:** Exposing specific CPU features can maximize the performance of workloads running inside the virtual machines.
* **Compatibility:** Ensuring consistent behavior across different hypervisor hosts in a virtualized cluster, regardless of the underlying host CPU, can be achieved by specifying a CPU model.
* **Migration:** CPU models play an important role in [Live Migration](/private-cloud-director/virtualized-clusters/virtualmachine/vm-migration#live-migration) and [Dynamic Resource Rebalancing Drr ](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr). Matching CPU models on source and destination hosts is crucial for successful live migration and for [Dynamic Resource Rebalancing Drr ](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr)to operate with full efficiency.
## Locating CPU Models Supported By Your Host
The libvirt KVM driver that comes built in as part of your Ubuntu installation on your hypervisor host provides a number of standard CPU model names.
These models are defined in `/usr/share/libvirt/cpu_map/*.xml`. You can inspect these files to determine which models are supported by your server.
Each file located under this directory contains information about the **feature set provided by the CPU model.**
For example, here the file `x86_SandyBridge-IBRS.xml`describes the CPU vendor, family, model and features supported by the SandyBridge-IBRS CPU Model.
{% tabs %}
{% tab title="Bash" %}
```bash
$ cat /usr/share/libvirt/cpu_map/x86_SandyBridge-IBRS.xml
...
```
{% endtab %}
{% endtabs %}
You can also locate the supported CPU models by running `virsh cpu-models ` command:
{% tabs %}
{% tab title="Bash" %}
```bash
$ virsh cpu-models x86_64
...
SandyBridge
SandyBridge-IBRS
IvyBridge
IvyBridge-IBRS
Haswell-noTSX
Haswell-noTSX-IBRS
Haswell
Haswell-IBRS
```
{% endtab %}
{% endtabs %}
## Supported CPU Models
space.vars.product\_name currently supports following list of [Supported Cpu Models List](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model/supported-cpu-models-list).
## CPU Model Configuration Per Host
When you configure a host with 'hypervisor' role, here are the steps followed by the compute service to configure the CPU model on the host:
1. It starts by checking the [list of approved models](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model/supported-cpu-models-list), beginning with the newest models based on the CPU vendor for your host.
2. For each model, it verifies compatibility by checking if the model is marked as **usable** in the `virsh -c qemu:///system domcapabilities` output.
3. The first approved model marked as usable is selected.
4. If no approved models are usable, the system falls back to a generic model.
## CPU Model Pre-requisites for Hypervisor Hosts
* Each host that you authorize in your space.vars.product\_name setup with 'hypervisor' role must support at least one CPU Model that is part of the [Supported Cpu Models List](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model/supported-cpu-models-list) list.
* Failing this requirement may result in unexpected errors such as the host going to offline state, or virtual machines failing to provision, etc.
## CPU Model Pre-requisites for Migration
Following CPU model prerequisites must be met by all hosts within a virtualized cluster, so that operations like live migration, cold migration, VM evacuation can work smoothly, and services like [Dynamic Resource Rebalancing Drr ](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr), [Virtual Machine High Availability Vm Ha ](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha)can operate efficiently.
* All hosts in the cluster must have the exact same value for the most recent CPU model supported. Read here for how [system configures CPU model per host](#cpu-model-configuration-per-host).
* This value must be part of the list of [Supported Cpu Models List](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model/supported-cpu-models-list).
* If the hosts do not all report the same most-recent supported CPU model, or if a host’s CPU model is missing from the supported models list above, then you can set the [CPU Mode and Model](/private-cloud-director/virtualized-clusters/virtualized-cluster#cpu-mode-and-model)
## Troubleshoot CPU Baseline Issues
If a host transitions to `error` or `offline` status after a host upgrade, or if live migrations begin failing with CPU compatibility errors, the cluster CPU baseline may need to be adjusted. See [Resolve CPU Baseline Mismatch After Host Upgrade](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/cpu-baseline-mismatch) for a step-by-step diagnostic and remediation guide, including how to use `cpu_model_extra_flags` for mixed-generation clusters.
# Supported CPU Models
This document provides a list of currently supported CPU Models in space.vars.product\_name.
## Supported Intel Models
## 2025+ Generation
```
ClearwaterForest
# 2024+ Generation
SierraForest SierraForest-v1
GraniteRapids GraniteRapids-v2 GraniteRapids-v1
# 2021-2023 Generation
SapphireRapids SapphireRapids-v3 SapphireRapids-v2 SapphireRapids-v1
# 2021 Generation
Snowridge Snowridge-v4 Snowridge-v3 Snowridge-v2 Snowridge-v1
# 2020 Generation
Cooperlake Cooperlake-v2 Cooperlake-v1
# 2019-2020 Generation
Icelake-Server Icelake-Server-noTSX Icelake-Server-v7 Icelake-Server-v6
Icelake-Server-v5 Icelake-Server-v4 Icelake-Server-v3 Icelake-Server-v2 Icelake-Server-v1
Icelake-Client Icelake-Client-noTSX
# 2017-2019 Generation
Cascadelake-Server Cascadelake-Server-noTSX Cascadelake-Server-v5
Cascadelake-Server-v4 Cascadelake-Server-v3 Cascadelake-Server-v2 Cascadelake-Server-v1
# 2015-2017 Generation
Skylake-Server Skylake-Server-IBRS Skylake-Server-noTSX-IBRS
Skylake-Server-v5 Skylake-Server-v4 Skylake-Server-v3 Skylake-Server-v2 Skylake-Server-v1
Skylake-Client Skylake-Client-IBRS Skylake-Client-noTSX-IBRS
Skylake-Client-v4 Skylake-Client-v3 Skylake-Client-v2 Skylake-Client-v1
# 2014-2015 Generation
Broadwell Broadwell-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS
Broadwell-v4 Broadwell-v3 Broadwell-v2 Broadwell-v1
# 2013-2014 Generation
Haswell Haswell-IBRS Haswell-noTSX Haswell-noTSX-IBRS
Haswell-v4 Haswell-v3 Haswell-v2 Haswell-v1
# 2012-2013 Generation
IvyBridge IvyBridge-IBRS IvyBridge-v2 IvyBridge-v1
# 2011-2012 Generation
SandyBridge SandyBridge-IBRS SandyBridge-v2 SandyBridge-v1
# 2010-2011 Generation
Westmere Westmere-IBRS Westmere-v2 Westmere-v1
# 2008-2010 Generation
Nehalem Nehalem-IBRS Nehalem-v2 Nehalem-v1
# 2007-2008 Generation
Penryn Penryn-v1
# 2006-2007 Generation
Conroe Conroe-v1
# Specialized processors
Denverton Denverton-v3 Denverton-v2 Denverton-v1
KnightsMill KnightsMill-v1
```
## Supported AMD Models
## 2022+ Generation
```
EPYC-Genoa EPYC-Genoa-v1
# 2021-2022 Generation
EPYC-Milan EPYC-Milan-v2 EPYC-Milan-v1
# 2019-2020 Generation
EPYC-Rome EPYC-Rome-v4 EPYC-Rome-v3 EPYC-Rome-v2 EPYC-Rome-v1
# 2017-2019 Generation
EPYC EPYC-IBPB EPYC-v4 EPYC-v3 EPYC-v2 EPYC-v1
# Legacy Opteron series (2003-2017)
Opteron_G5 Opteron_G5-v1
Opteron_G4 Opteron_G4-v1
Opteron_G3 Opteron_G3-v1
Opteron_G2 Opteron_G2-v1
Opteron_G1 Opteron_G1-v1
# Chinese Zen-based processors
Dhyana Dhyana-v2 Dhyana-v1
# Legacy consumer processors
phenom phenom-v1
```
## Supported Generic Models
## KVM virtualization models
```
qemu64 qemu64-v1
kvm64 kvm64-v1
# RHEL/CentOS compatibility models
cpu64-rhel6
cpu64-rhel5
```
# Manually Configure CPU Model
Follow this document to manually configure CPU model for all hypervisor hosts in your virtualized cluster.
Occasionally you may need to manually configure the CPU model across hosts in your virtualized cluster, to ensure consistent behavior for important features such as [Dynamic Resource Rebalancing DRR ](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr), [Virtual Machine High Availability Vm Ha ](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha), [Maintenance Mode](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/maintenance-mode) and overall predictable performance across your hypervisor hosts.
Follow the steps to do this:
## Step 1: Query the existing CPU model on a host
Run the following command to query the list of supported CPU models on a host
{% tabs %}
{% tab title="Bash" %}
```bash
$ virsh domcapabilities | grep "model usable='yes" | sort
```
{% endtab %}
{% endtabs %}
This command will show output similar to the following:
{% tabs %}
{% tab title="Bash" %}
```bash
486Broadwell-noTSX-IBRSBroadwell-noTSXCascadelake-Server-noTSXConroecore2duocoreduoHaswell-noTSX-IBRSHaswell-noTSXIvyBridge-IBRSIvyBridgekvm32kvm64n270Nehalem-IBRSNehalemOpteron_G1Penrynpentium2pentium3pentiumqemu32SandyBridge-IBRSSandyBridgeSkylake-Client-noTSX-IBRSSkylake-Server-noTSX-IBRSWestmere-IBRSWestmere
```
{% endtab %}
{% endtabs %}
## Step 2 - Choose a Common CPU Model
Run the above command on all hypervisor hosts in your virtualized cluster to find the list of supported models. Then choose the most recent CPU Model that is supported across all hosts.
## Step 3 - Apply CPU Model Changes to All Hosts
Now that you have identified the common model, make changes to the compute service config file to apply the CPU model to all hypervisor hosts in your virtualized cluster.
Follow [CPU Mode and Model Configuration](/private-cloud-director/virtualized-clusters/nova-override#cpu-mode-and-model-configuration) and edit the CPU model value with the model you identified in step 2.
Make sure to [Restart Compute Service](/private-cloud-director/virtualized-clusters/nova-override#restart-service) on each host following the changes.
# Advanced Remote Support
Advanced Remote Support (ARS) is a secure troubleshooting mechanism that allows Platform9 support engineers to log onto your space.vars.product\_name hosts to analyze and resolve complex technical issues. By default, Platform9 support team members cannot interactively access your hosts. However, when you enable ARS, support engineers can securely connect through the host's existing connection to the Platform9 management plane.
Despite being based on SSH, ARS does not expose your host to SSH login from any external network. The mechanism leverages secure channels and does not require any firewall changes to your host or network infrastructure. In this guide, you will learn how to enable, configure, and disable Advanced Remote Support for your space.vars.product\_acronym hosts.
## Prerequisites
Before you enable Advanced Remote Support, ensure you have:
* Administrative access to the space.vars.product\_acronym UI
* Root or sudo access to the target host
* The SSH daemon (`sshd`) running on the target host
* The `pf9` user account created on the host (this is created automatically during space.vars.product\_acronym installation)
## Enable Advanced Remote Support
To enable Advanced Remote Support for a host:
1. Navigate to **Infrastructure → Cluster Hosts** in the space.vars.product\_acronym UI.
2. Select the checkbox next to the host where you want to enable remote support.
3. Click **Edit Roles**.
4. Select the **Advanced Remote Support** checkbox.
5. Click **Update Role Assignment**.
**Expected outcome:** The host now allows Platform9 support engineers to establish secure SSH connections through the management plane.
## Verify SSH Daemon Configuration
To ensure the SSH daemon is properly configured:
1. Connect to your host using your standard SSH method.
2. Verify that the `sshd` service is running by executing the appropriate command for your operating system.
3. Confirm that the SSH daemon configuration allows key-based authentication.
Consult your Linux operating system's documentation for specific instructions on verifying and configuring the SSH daemon.
## Configure Sudo Access (Optional, Recommended)
When ARS is enabled, Platform9 support engineers log into the host using the `pf9` user account. By default, this account has restricted privileges. To allow support engineers to run diagnostic commands with elevated privileges, you need to configure sudo access for the `pf9` user.
### Requirements for Sudo Access
To grant sudo access:
* sudo must be enabled for the 'pf9' user
* sudo must allow the 'pf9' user to authenticate without a password (ARS uses one-time SSH keys, so the 'pf9' user does not have a password by default)
### Configure Sudo on Debian and Ubuntu Systems
To configure sudo access on Debian-based systems:
1. Edit the sudo rules by running the visudo command.
2. Add or verify the following line to allow wheel group members to authenticate without a password:
{% tabs %}
{% tab title="Bash" %}
```bash
pf9 ALL=(ALL) NOPASSWD: ALL
```
{% endtab %}
{% endtabs %}
3. Save and exit the editor.
**Expected outcome:** The 'pf9' user can now run commands with sudo privileges without entering a password.
For other Linux distributions, consult your operating system's documentation for specific instructions on configuring passwordless sudo access.
## Coordinate with Platform9 Support
After enabling Advanced Remote Support, coordinate with your Platform9 support representative to arrange access:
1. Identify the specific host that requires troubleshooting by sharing:
* The contents of the host's `/etc/pf9/host_id.conf` file, or
* The host's hostname
2. Agree on a time window when the support engineer can access the host.
3. Confirm that ARS is enabled and properly configured on the target host.
## Disable Advanced Remote Support
When troubleshooting is complete, you should disable Advanced Remote Support to restore standard access controls.
To disable Advanced Remote Support:
1. Navigate to **Infrastructure → Cluster Hosts** in the space.vars.product\_name UI.
2. Select the checkbox next to the host where you want to disable remote support.
3. Click **Edit Roles**.
4. Deselect the **Advanced Remote Support** checkbox.
5. Click **Update Role Assignment**.
**Expected outcome:** Platform9 support engineers can no longer establish SSH connections to the host through the management plane.
# Host Aggregate
A host aggregate is a **group of hosts** within your virtualized cluster that share common characteristics. Host aggregates are a mechanism to create specialized groupings of hosts that have specific properties, with the purpose of enabling provisioning of VMs that require those properties on those hosts.
You can create a host aggregate by navigating to the space.vars.product\_name UI and selecting 'Infrastructure' -> 'Host Aggregates' from the left nav bar. When you create a new host aggregate, you get to specify a name for the aggregate and a set of key-value pairs that describe the specific properties of the hosts that will belong to this host aggregate.
For example, you may wish to create a group of hosts within your virtualized cluster that have Oracle licensing enabled on them. You can do this by creating an aggregate called "Oracle Hosts" and within that aggregate, creating a key-value pair with values "Oracle-License" and "True". You can then add the hosts that you know have Oracle license installed on them to this aggregate.
This will enable you to provision VMs that require Oracle licensing to place on these hosts. You will do this by first creating a flavor that references this host aggregate and then using that flavor to create your VM. Read about [Flavors and Host Aggregates](/private-cloud-director/virtualized-clusters/virtualmachine#flavors-and-host-aggregates) for how to do this.
# Virtual Machines
A virtual machine, or a VM, is a software-based representation of a physical computer. A VM allows you to run an operating system and applications using the resources of a host machine or a hypervisor, acting like a separate, isolated computer with its own virtual CPU, memory, storage, and network capabilities, all managed by the hypervisor.
You can create a virtual machine by first [populating an image in the image library](/private-cloud-director/images-and-image-library/image-library---images), then creating one or more VM flavors, and then creating a new VM.
## VM Flavors
Once you have the images you would like to use, it's time to look at the resource configuration for the VMs to be deployed. space.vars.product\_name uses T-shirt-sized configurations of resource allocations, called Flavors, to allow you to specify the resource allocation for VMs. Read more about [VM Flavors](/private-cloud-director/virtualized-clusters/virtualmachine/vm-flavors) here.
## VM Affinity and Anti-affinity Rules space.vars.product\_name supports the creation of VM affinity and anti-affinity groups using a mechanism called Server Groups. Read more about [VM Affinity Anti-Affinity Rules](/private-cloud-director/virtualized-clusters/virtualmachine/vm-affinity-anti-affinity-rules) here.
## Create a VM
By now, you have your desired VM image, your preferred Flavor, and you have previously setup at least one Network that you can use to deploy your VM. So navigate to Virtual Machines in the Navigation pane and start 'Deploy Virtual Machine'.
We will describe some of the VM creation options here.
### VM source
{% hint style="warning" %}
**Warning**
This is an important selection that affects where your virtual machine disk is stored, and whether the VM will be using [Ephemeral Storage](/private-cloud-director/storage/ephemeral-storage) or [Block Storage](/private-cloud-director/storage/block-storage)
{% endhint %}
#### Boot VM from Image
Use this option to provision a VM that uses [Ephemeral Storage](/private-cloud-director/storage/ephemeral-storage) for its root disk. This storage option is typically used for non-production VMs where the disk need not be preserved across host failures or VM deletion. The VM root disk will be created using local storage on the hypervisor on which it is provisioned and populated with the contents of the source image. If the hypervisor were to go down, it would not be possible to access or recover the VM disk till the hypervisor comes back online. If the host's local disk experiences corruption or other issues, the VM may be impacted and may be unrecoverable. If the VM were to be deleted, the VM's root disk would be deleted along with it and would not be recoverable afterward.
#### Boot VM from New Volume
Use this option if you would like to create a VM that uses [Block Storage](/private-cloud-director/storage/block-storage) volume for its root disk. A VM created in this manner will persist across host failures and VM termination, unless you choose to delete the volume when the VM terminates. When you choose this option, a new persistent volume is created on the block storage you configured for your virtualized cluster, and its contents are populated from the source image.
#### Boot VM from Existing Volume
This option is like the previous one, except that you choose to boot from an existing volume rather than provisioning a new volume from a base image.
#### Install OS from ISO Wizard
You can now create Windows and Linux VMs directly from installation ISOs through the **Deploy New VM** wizard in the UI. This replaces the previous CLI-only workflow documented in the [Windows](/private-cloud-director/tutorials/create-windows-vm-from-iso/creating-a-windows-virtual-machine--vm--from-an-iso-image4io) and [Ubuntu](/private-cloud-director/tutorials/deploy-a-virtual-machine-using-an-ubuntu-iso-image) tutorials.
**To install an OS from an ISO:**
1. In the **Deploy New VM** wizard, select **Install from ISO** from the **Boot VM From** dropdown.
2. Select an installation ISO from your image library and set the **Root Volume Size**. A new volume of the specified size will be created for the OS installation.
3. (Optional) For Windows installs, enable **Drivers ISO** and select the VirtIO drivers ISO for optimal performance.
4. Complete the remaining steps (optionally attach additional volumes, select a flavor, configure networking, customize VM) and click **Deploy Virtual Machine**.
5. Connect to the VM console and complete the OS installation. The VM is configured with a boot order of boot volume first, then ISO, so once the OS installer finishes and the VM reboots, it will automatically boot from the installed disk.
**Notes:**
* Only zero-disk flavors are supported, since the root disk is provided by the volume created in step 2. The VM disk size will equal the volume size.
* Bulk VM creation is not supported when installing from an ISO.
* The installation ISO can remain attached to the VM after installation. It will not interfere with boot because of the boot order, and it remains in the image library. You can detach it manually if you prefer.
#### Guest OS hostname
By default, the value you enter in the **Name** field is used both as the VM's display name in Private Cloud Director and as the guest OS hostname. To set a hostname that differs from the VM name, use the **Guest OS Hostname (Optional)** field in the **Basic VM Info** section. This is useful when you want a descriptive VM name for tracking and reporting, paired with a shorter, standards-compliant hostname inside the OS.
If you leave the **Guest OS Hostname (Optional)** empty, the **VM** **Name** is used as the guest OS hostname.
When you deploy more than one VM in a single operation, Private Cloud Director appends a numeric suffix to the hostname so that each VM has a unique name, using the same pattern as VM names.
If you set a value in **Guest OS Hostname (Optional)** and also include a cloud-init script that sets the hostname, the cloud-init script takes precedence. Use only one method to set the hostname to avoid unexpected behavior.
### Choose a Specific Subnet
When the network you select has more than one subnet, the **Networking** step of the **Deploy New VM** wizard displays a **Subnet** column for each network row. The selector defaults to **Automatic**; expanding it lists each available subnet as ` ()` — for example, `192.0.0.0/5 (test)`. Select one to place the VM's network interface on that specific subnet. If you leave it on **Automatic**, the Networking Service chooses the subnet for you.
Choosing a specific subnet does not limit how many VMs you can create in a single operation. If, however, you switch a network's IP mode from **Automatic** to **Private IP** and enter a specific address, the wizard's **Customize VM** step restricts you to a single VM and shows:
> You can create only 1 VM when a specific port is selected from a network.
For dual-stack or other advanced port configurations, pre-create a port on the subnet you want and attach the port during the **Networking** step. See [Virtual Network](/private-cloud-director/virtualized-networking/networks-and-ports) and [Add Multiple Dual Stack Ports to a VM](/private-cloud-director/tutorials/create-dual-stack-port).
### SSH Key
Using SSH Keys is the recommended and secure method to access your VMs. To set up your SSH Key, navigate to 'Networks and Security' in the Navigation Pane, and import your SSH Keys.
Once you have the key imported, you can select that key when creating a new VM.
### Select Server Group
Optionally select an [affinity or anti-affinity group](#vm-affinity-and-anti-affinity-rules) for this VM to be part of. This will impact the VM's placement on a host in your virtualized cluster.
### Using cloud-init
Using cloud-init allows you to customize the virtual machine as it boots, such as provisioning additional software or custom scripts.
### Assign Security Groups
Security groups allow you to limit port access to Virtual Machines. If you don't already have a Security Group configured, you can cover this later.
### Specify Metadata
Metadata attributes allow you to specify additional key-value pairs, which can be used to query and organize your virtual machines.
{% hint style="success" %}
**Success**
With this information specified, you should see the VM provision within a few minutes. You can then access the VM over the network or via its console.
{% endhint %}
## VM Properties
### VM UUID
Each newly created virtual machine is assigned a unique UUID. By default, this field is not displayed in the virtual machines grid view in the UI. You can change this by clicking the 'Manage Columns' button above the virtual machines grid view and selecting the UUID field. You can also select an individual VM to open the VM details view and see the ID property listed there. Alternatively, you can get VM ID using the pcdctl CLI by running the `pcdctl server list` command or the `pcdctl server show` command and supplying the name of the VM.
## Migrate Existing VMs from VMware
If you have existing workloads that you would like to migrate onto your space.vars.product\_name cluster, [Project vJailbreak](https://github.com/platform9/vjailbreak) can help.
### Download the vJailbreak Appliance
Head over to Project vJailbreak to download the appliance image. The appliance is packaged as an OVA that you can deploy into your space.vars.product\_name environment. Note that vJailbreak should have access to your storage network to efficiently migrate VM data over during the data copy phase.
### Register your Source VMware Environment and Your New Virtualized Cluster
As your vJailbreak appliance loads, you'll see instructions for accessing its user interface, where you can specify your source VMware environment information, as well as the information for your new cluster.
### Select VMs and Schedule Migrations
With this information specified, you can now initiate migrations from the vJailbreak user interface.
# VM Flavors
space.vars.product\_name uses t-shirt sized configurations of resource allocations, called flavors, to allow you to specify the resource allocation for VMs. This feature enables administrators to templatize and standardize on a few different VM configurations that developers can use, rather than having a configuration sprawl.
space.vars.product\_name ships with a set of flavors out of box, but administrators can create new flavors and make them available to end users.
## Flavor Properties
Flavors capture the following configuration information for a VM:
1. vCPUs - number of CPU cores
2. RAM - amount of memory
3. Disk - size of the VM ephemeral disk, **only used when** creating a VM from an image or a [VM Ephemeral Disk Snapshot](/private-cloud-director/virtualized-clusters/virtualmachine/virtual-machine-snapshot#type-1---vm-ephemeral-disk-snapshot). Ignored when creating a VM using a volume for root disk.
4. Metadata - used to define:
1. What [Host Aggregate](/private-cloud-director/virtualized-clusters/host-aggregate) the VM should get placed on, if any, by specifying metadata (key-value pairs) that correspond to a specific [Host Aggregate](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster#host-aggregates). When creating a VM using this flavor, the VM will be placed on hosts in your virtualized cluster that are part the selected host aggregate (that match the specified metadata attributes).
## Create Flavors
Navigate to the Flavors menu in the UI under Virtual Machines, and create or edit flavors to suit your desired CPU, memory and disk configuration.
## Zero CPU or Memory Flavors
space.vars.product\_name allows you to create flavorless VMs where you specify CPU and memory information dynamically at VM creation time, instead of having to use a pre-define value via a flavor. This is done by using special flavors that have both CPU and memory set to zero. Read more about creating [flavorless VMs here](/private-cloud-director/virtualized-clusters/virtualmachine/flavorless-vms-with-hot-plug).
## Zero Disk Flavors
You can create a VM flavor with disk size set to zero.
Using such a flavor is a requirement when creating a VM that will use a block storage volume for it's root disk.
When used to create a VM that will use ephemeral root disk, the root disk size for the VM will be set to be equal to the size of the image.
# VM Affinity Anti-Affinity Rules
This document describes how to create affinity and anti-affinity rules for your virtual machines, the placement behavior each rule produces, and how those rules interact with other space.vars.product\_name operations such as Dynamic Resource Rebalancing (DRR), VM High Availability (VM HA), maintenance mode, and migration.
## Overview
Server groups are the mechanism space.vars.product\_name uses to apply affinity and anti-affinity rules. A server group is a logical group of virtual machines that shares a single placement policy. Membership in a server group controls how the member VMs are placed across the hypervisor hosts in your virtualized cluster.
space.vars.product\_name supports four policies. Two are strict ("hard") and two are best effort ("soft").
| Policy | Rule strength | Placement goal | If the goal cannot be met |
| ------------------ | ------------- | ---------------------------------------------- | ------------------------------------ |
| Affinity | Strict (hard) | All VMs in the group run on the same host | The operation fails |
| Soft affinity | Best effort | Keep all VMs in the group on the same host | VMs may be placed on different hosts |
| Anti-affinity | Strict (hard) | No two VMs in the group run on the same host | The operation fails |
| Soft anti-affinity | Best effort | Spread VMs in the group across different hosts | Two or more VMs may share a host |
* **Affinity (hard).** Restricts all VMs in the group to the same hypervisor host. If no single host can hold every member, the operation fails.
* **Soft affinity.** Attempts to keep all VMs in the group on the same host, but allows them to run on different hosts when that is not possible.
* **Anti-affinity (hard).** Restricts the VMs in the group to separate hosts, so that no two members run on the same host. If there are not enough eligible hosts, the operation fails.
* **Soft anti-affinity.** Attempts to keep the VMs in the group on separate hosts, but allows two or more of them to share a host when that is not possible.
A server group has exactly one policy. You cannot combine affinity and anti-affinity, or hard and soft, within the same group.
## Prerequisites
* A virtualized cluster with one or more hosts that have the hypervisor role.
* Enough host capacity for the policy you choose. A hard affinity group needs a single host with enough capacity for every member. A hard anti-affinity group needs at least as many eligible hosts as it has VMs. Anti-affinity and soft anti-affinity are only meaningful when the cluster has more than one hypervisor host.
* The server group must exist before you create the VMs that will belong to it.
* The appropriate role. Administrators and self-service users can create and delete server groups. Read-only users cannot.
## Create a Server Group
To create a server group from the UI:
1. Navigate to **Virtual Machines** > **Server Groups** and click the **Create Server Group** button.
2. In the **Name** field, enter a descriptive name for the server group.
3. From the **Policy** dropdown, select one of **Affinity**, **Soft Affinity**, **Anti-Affinity**, or **Soft Anti-Affinity**. See [Overview](#overview) for what each policy does.
4. Click the **Create Server Group** button.
You can now add a VM to this server group as part of the VM creation wizard.
## Add a VM to a Server Group
A VM's server group is selected when the VM is created, in the VM creation wizard.
You cannot add an existing VM to a server group, and you cannot move a VM from one server group to another, after the VM has been created. To change membership, recreate the VM in the desired group. You can create the new VM from an image of the original VM to preserve its disk contents.
## How Affinity Rules Interact With space.vars.product\_name Operations
After a server group is created and its VMs are running, the group's policy continues to govern placement during automated and administrative operations. The table below summarizes the behavior for powered-on VMs.
| Policy | DRR | VM HA | Maintenance mode | Live and cold migration |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Affinity** (hard) | DRR does not rebalance members of a hard affinity group. The Compute Service cannot find a valid target for a single member without breaking the rule, so the VM is skipped. | On host failure, VM HA looks for a single host with enough capacity for the entire group and restarts all members there together. If no such host exists, HA fails for the group and the UI reports the failure. | Entering maintenance mode fails if any powered-on member of a hard affinity group is on the host. The UI reports the policy violation. | Migration fails. The platform cannot move every member together in a single operation, so a single member cannot be relocated without breaking the rule. See [Limitations](#limitations). |
| **Soft affinity** | DRR may rebalance members. A member with no explicit migration priority is treated as low priority for rebalancing; a member with a set migration priority is honored at that priority. | VM HA tries to restart all members together on a single host with enough capacity. If none is available, it restarts the members individually, which may place them on different hosts, and the UI reports an affinity policy violation. | Best effort. space.vars.product\_name keeps members together where capacity allows and places the remainder on other hosts. The group may end up split across hosts. | Allowed. The UI warns that the move may violate the soft affinity policy but does not block it. |
| **Anti-affinity** (hard) | When choosing a migration target, DRR excludes any host that already runs another member of the group. | On host failure, VM HA restarts each VM on a host that is not running another member VM. If no compliant host is available, HA fails for that VM and the UI reports the failure. A member on an offline host is shown as disconnected. | Entering maintenance mode fails if no target host preserves the anti-affinity rule. | Migration fails when the selected target host already runs another member of the group. The UI filters out non-compliant hosts. |
| **Soft anti-affinity** | Same rebalancing behavior as soft affinity. Members are honored at their set migration priority, or treated as low priority when none is set. | VM HA tries to restart each member on a host that is not running another member. If none is available, it may restart on a host that already runs a member and reports an anti-affinity policy violation. If it cannot restart the VM at all, HA fails and the VM is shown as disconnected. | Best effort. Members may be co-located on a host when capacity requires it. | Allowed even when the target host runs another member. The UI warns of the membership but does not block the move. |
{% hint style="info" %}
**Hot add of CPU or memory does not affect affinity rules.** A hot add operation does not move a VM to a different host, so a hot-added VM keeps the same placement and never violates an affinity or anti-affinity rule, regardless of the policy type.
{% endhint %}
## Limitations
* space.vars.product\_name cannot move all members of a server group together in a single, atomic operation. As a result:
* Live or cold migration of a VM in a **hard affinity** group fails. Moving one member VM on its own would break the rule that all members share a host, and the platform cannot relocate the entire group in one step.
* Live or cold migration of a VM in a **hard anti-affinity** group fails when the selected target host already runs another member of the group.
* Soft affinity and soft anti-affinity groups are not affected, because their rules are best effort and the platform is allowed to break them when necessary.
* **Group membership is set at VM creation only.** You select a VM's server group when you create the VM. You cannot add an existing VM to a group, or move a VM between groups, afterward.
* **One policy per group.** A server group enforces a single policy. Affinity and anti-affinity, or hard and soft, cannot be mixed in the same group.
* **Capacity and host count govern strict policies.** A hard affinity group needs one host with enough capacity for every member, and a hard anti-affinity group needs at least as many eligible hosts as it has members. When these conditions are not met, VM creation, an HA restart, or a maintenance-mode evacuation can fail because no valid host is available.
## VMware Equivalents
If you are coming from VMware, space.vars.product\_name affinity rules map to vSphere DRS affinity rules as follows.
Private Cloud Director policy
Closest VMware DRS rule
Affinity
"Keep Virtual Machines Together", or a VM-Host "Must run on hosts in group" rule
Soft affinity
A preferential "should run" affinity rule
Anti-affinity
"Separate Virtual Machines", or a VM-Host "Must not run on hosts in group" rule
Soft anti-affinity
A preferential "should not run" separation rule
As in vSphere, a hard rule blocks an HA restart that would violate it, while a soft rule lets HA restore availability first, even if that temporarily breaks the rule, and rebalances toward compliance afterward.
# Virtual Machine Actions
### Rebuild VM
The rebuild operation allows for the re-creation of virtual machine from a new or existing image while preserving the VM's:
* UUID
* Storage Volumes
* IP addresses (private or public)
* Network ports
If the VM utilizes ephemeral storage (local disks), rebuilding will result in the loss of data on those disks. However, if the VM uses block storage volumes or shared storage for its root disk and other data, those volumes are reattached to the rebuilt virtual machine, and data should be preserved.
### Rescue / Unrescue VM
Rescue mode is a powerful operation that allows you to access and repair a non-bootable virtual machine. When activated, this action boots your virtual machine into a temporary environment using the image that was used to create the vm, with full root access to the file system. This mode is useful for:
* Troubleshooting and fixing configuration file issues
* Recovering or copying data to a remote location
* Gaining emergency access similar to single-user mode or safe mode with networking
You can rescue a virtual machine using a new or different image from the one used to create it. When you select this option in the UI, the UI shows a list of images from the image catalog, so you can choose a different image to boot from.
To enable an image to be used for rescuing VMs, add the following metadata keys to the image:
* **`hw_rescue_device`** – The type of device to attach the rescue image as (`cdrom`, `disk`, or `floppy`)
* **`hw_rescue_bus`** – The bus the rescue device should use (`scsi`, `virtio`, `ide`, or `usb`)
For a typical VM qcow2 image to be used as rescue image, the values you would use would be `hw_rescue_device: disk` and `hw_rescue_bus: scsi.`
Once your maintenance or recovery tasks are complete, you can return the VM to normal operation by unrescuing the VM. To do that using the UI, select the VM in the VM grid view, then choose the `unrescue` action from the power actions drop down menu.
### Stop
Performs a graceful shutdown of the virtual machine by sending an ACPI shutdown signal to the guest operating system. The VM will attempt to shut down cleanly, allowing running processes to terminate properly and data to be flushed to disk. If the guest OS does not respond to the ACPI signal, the VM may not shut down. In such cases, use Hard Reboot instead.
### Reboot
Performs a graceful restart of the virtual machine by sending an ACPI reboot signal to the guest operating system. This allows the OS to restart cleanly, preserving data integrity.
### Hard Reboot
Forces an immediate restart of the virtual machine without waiting for the guest OS to shut down cleanly. This is equivalent to pressing the physical reset button on a server and immediately powers the VM back on. This action may result in data loss or corruption as running applications do not have time to shut down properly. Use only when a normal reboot fails or the VM is unresponsive.
### Suspend
Saves the current running state of the virtual machine to disk, including memory contents and CPU state, then stops the VM. When resumed, the VM will continue exactly where it left off, with all applications and data in the same state.
### Pause
Freezes the virtual machine execution by halting the virtual CPU without writing the state to disk. The VM's memory remains allocated in RAM, but no CPU cycles are consumed. This is a temporary state intended for short pauses.
### Rename
Changes the display name of the virtual machine. This updates only the VM's label in the space.vars.product\_name interface and does not affect the guest OS hostname or network identity.
### Change Owner
Transfers ownership of the virtual machine to a different user account. The new owner will have full control over the VM and its associated resources. Please ensure that the new owner has available quota to take ownership of the VM before transferring ownership.
### Edit Metadata
Allows you to add, modify, or remove custom key-value metadata tags associated with the virtual machine.
### Resize
Allows you to choose a new flavor to resize the VM to. Resizing a VM results in the VM being rebuilt (powered off, then recreated using the source image on the same or different host) using the new flavor. Please note that the disk size of the new flavor must be larger than or equal to the current flavor.
### Clone
Creates a complete copy of the virtual machine, including its disks and configuration. The new VM is independent of the original and can be modified without affecting the source VM.
### Migration Priority
Sets the priority level for VM migrations triggered by Dynamic Resource Rebalancing (DRR). Higher priority migrations are processed first during automated resource rebalancing or operations. You can specify migration priority as Low, Normal, High, or Excluded. If a migration priority is not set for a VM, it defaults to Normal priority. Selecting Excluded excludes the VM from any DRR migrations.
### Mount ISO
Attaches an ISO from the image library to a VM in a powered-off state. This action presents the mounted ISO as a read-only optical drive to the guest OS.
### Unmount ISO
Detaches an ISO that was previously attached to the VM. This action requires the VM to be powered off before the ISO can be detached.
# Virtual Machine Migration
Discover VM migration in PCD, a vital process for moving virtual machines between hosts to ensure system availability and optimize resource utilization. Learn about use cases, types, and commands for
VM migration in space.vars.product\_name refers to the process of moving a running or stopped virtual machine (VM) from one compute host (hypervisor) to another within the same space.vars.product\_name environment. This is a fundamental capability for maintaining system availability, performing infrastructure maintenance, and balancing workloads in data centers.
## Common Use Cases
Following are the common scenarios in which migration is initiated in space.vars.product\_name.
* **Maintenance mode:** When you invoke [Maintenance Mode](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/maintenance-mode) on host, VMs are live migrated to other compatible hosts in the virtualized cluster.
* **Load Balancing:** space.vars.product\_name [Dynamic Resource Rebalancing (DRR)](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr) uses live migration to distribute VMs more evenly across hosts in a virtualized cluster to optimize resource utilization and prevent individual hosts from becoming overloaded.
* **Host Failure (VM Evacuation):** space.vars.product\_name [Virtual Machine High Availability (VM HA)](/private-cloud-director/virtualized-clusters/virtualized-cluster/virtual-machine-high-availability-vm-ha) service uses VM evacuation to recover VMs from a failing or failed host.
## Types of VM Migration
space.vars.product\_name supports following types of VM migrations:
* Cold migration
* Live migration
* VM evacuation
Let's dive into the specifics of each.
## Cold Migration
Cold migration is a method of moving a virtual machine from one host to another, where the VM is `shut down`` ``(powered off)`or `suspended` during the process. The VM disks and configuration files are transferred to the destination host as part of the cold migration process.
**Downtime:** Cold migration essentially requires VM downtime, as the VM must be powered off or suspended before it can be cold migrated.
### Cold Migration Prerequisites
* VM should be powered off or suspended.
* Cold migration **uses the management network** for copying over virtual machine disk information. Make sure that the management network interface has **sufficient network bandwidth** **and capacity** to support **disk copy operation**.
* All hosts within the virtualized cluster must meet [CPU Model Pre-requisites](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model#cpu-model-pre-requisites).
* Source and destination hosts must run compatible operating systems. Cold migration is supported between hosts running the same Ubuntu version, **and from Ubuntu hosts to Platform9 OS powered by Rocky Linux by CIQ hosts** (the supported path for transitioning to Platform9 OS). For more information, refer to the [Operating System Compatibility Requirements](#operating-system-compatibility-requirements).
#### Using CLI
1. Authenticate [pcdctl CLI](/private-cloud-director/reference/pcdctl-command-line)
2. Run the command below to perform the cold migration. The command below will cold migrate the VM to a suitable host within the virtualized cluster.
{% tabs %}
{% tab title="Bash" %}
```bash
$ pcdctl server migrate
```
{% endtab %}
{% endtabs %}
3. To migrate VM to a specific host use the below command:
{% tabs %}
{% tab title="BASH" %}
```bash
$ pcdctl server migrate --host
```
{% endtab %}
{% endtabs %}
#### Using UI
* Log in to the UI and select Virtual Machine menu on the left side nav bar.
* Select the desired VM then click on the "other" option on the action bar and select the `migrate` action.
* Select the target hypervisor host and hit the "Migrate VM" button.
## Live Migration
Live migration is a method of moving a virtual machine from one host to another, where the VM remains **running** throughout the migration process.
The live migration process copies a virtual machine's memory from the source to the destination host while the VM is running. Any memory pages that get modified or "dirtied" on the VM during this time are copied over again. Finally, the VM enters a brief pause period during which its remaining memory and CPU state are copied over to the destination host, and finally, the VM is resumed on the destination host.
**Downtime:** The VM will experience a brief downtime (typically milliseconds to seconds) during the pause period before resuming on the destination host.
Live migration can be classified further by the way it treats virtual machine storage:
* **Live Migration for a VM using Ephemeral Shared Storage:** The virtual machine has an ephemeral root disk that is located on [Ephemeral Shared Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-shared-storage).
* Live migration enables the testing and validation of whether the source and target hosts are utilizing the same underlying shared storage.
* When validated, live migration is performed without copying over VM disk.
* **Live Migration of a VM using Ephemeral Local Storage:** The Virtual machine has an ephemeral root disk that is located on [Ephemeral Local Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-local-storage).
* Live migration will copy over the entire virtual machine root disk from the source host to the destination host.
* **Live Migration of a VM using Volumes only:** The Virtual machine is using [block storage volumes](/private-cloud-director/storage/volume), rather than ephemeral disk.
* Live migration will migrate the VM without performing a storage copy. The same volumes will continue to be attached to the VM once it's migrated to the target host.
### Live Migration Prerequisites
* All hosts within the virtualized cluster must run the same major and minor version of Ubuntu.
* Live migration uses the **management network to copy virtual machine memory** and, optionally, disk information by default. Make sure that the management network interface has **sufficient network bandwidth** **and capacity** to support **VM memory migration traffic**. A separate network interface can be configured for live migration traffic to isolate it from the management network.
* When using live migration in a production setup, specifically for features like [Dynamic Resource Rebalancing (DRR)](/private-cloud-director/virtualized-clusters/virtualized-cluster/dynamic-resource-rebalancing-drr), it is recommended to configure all virtual machines to use **shared storage only** for the best performance of the migration operation.
* When using [Ephemeral Local Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-local-storage), live migration requires **sufficient network bandwidth and capacity** to **support virtual machine disk copy**.
* This is not required when using [Ephemeral Shared Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-shared-storage) at the virtualized cluster level or using [Block Storage Volumes](/private-cloud-director/storage/volume) exclusively for all virtual machines.
* All hosts within the virtualized cluster must meet [CPU Model Pre-requisites](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model#cpu-model-pre-requisites) for live migration to work. Read more about [CPU Models](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model).
* Source and destination hosts must run the same operating system version. For more information, refer to the [Operating System Compatibility Requirements](#operating-system-compatibility-requirements).
{% hint style="warning" %}
**WARNING**
Live migration between hosts with different Ubuntu versions (22.04 and 24.04) is not supported and will result in migration failures. Ensure all hosts in your virtualized cluster run the same operating system version.
{% endhint %}
#### Live Migrate Using CLI
1. Authenticate [pcdctl CLI](/private-cloud-director/reference/pcdctl-command-line)
2. Run the command below to perform the live migration.
{% hint style="info" %}
**NOTE**
Here target host UUID is mandatory.
{% endhint %}
{% tabs %}
{% tab title="Bash" %}
```bash
pcdctl server migrate --live-migration --host
```
{% endtab %}
{% endtabs %}
#### Live Migrate Using UI
1. Log in to the space.vars.product\_acronym UI and select **Virtual Machine**.
2. On Virtual Machine search for a specific VM and select the VM.
3. Click **other** tab and then select the **Migrate**.
This will list down the eligible target hypervisors for Live migration.
4. Select the target hypervisor and click **Migrate VM**.
## VM Evacuation
VM Evacuation is specifically designed for disaster recovery. If a host fails or becomes unresponsive, VM evacuation attempts to restart a VM that was running on the failed host onto a healthy host.
### VM Evacuation Prerequisites
* VM evacuation can only be performed for VMs using [Ephemeral Shared Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-shared-storage) or [Block Storage Volumes](/private-cloud-director/storage/volume). If the VM is using [Ephemeral Local Storage](/private-cloud-director/storage/ephemeral-storage#ephemeral-local-storage), evacuation can not work as the virtual machine disk can not be copied over from the source host to a target host.
* All hosts within the virtualized cluster must meet [CPU Model Pre-requisites](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model#cpu-model-pre-requisites) for VM evacuation to work. Read more about [CPU Models](/private-cloud-director/virtualized-clusters/add-hosts-virtualized-cluster/cpu-model).
* Source and destination hosts must run the same operating system version. For more information, refer to the [Operating System Compatibility Requirements](#operating-system-compatibility-requirements).
{% hint style="info" %}
**NOTE**
The evacuation operation will be triggered automatically for VMs on hosts that experience an outage in clusters with VM High Availability enabled. You can perform a VM evacuation operation manually via the command line.
{% endhint %}
#### Using CLI
1. Authenticate [pcdctl CLI](/private-cloud-director/reference/pcdctl-command-line)
2. Run the command below to perform the live migration.
{% hint style="info" %}
**NOTE**
Here target host UUID is mandatory.
{% endhint %}
{% tabs %}
{% tab title="Bash" %}
```bash
$ pcdctl server evacuate --host
```
{% endtab %}
{% endtabs %}
### Operating System Compatibility Requirements
{% hint style="info" %}
**NOTE**
Live migration and VM evacuation require source and destination hosts to run the same operating system version and KVM version. **Cold migration** additionally supports a one-way Ubuntu-to-Platform9-OS path; see [Cross-OS Cold Migration: Ubuntu Hosts to Platform9 OS Hosts](#cross-os-cold-migration-ubuntu-hosts-to-platform9-os-hosts) below.
{% endhint %}
* Hosts running Ubuntu 22.04 can only migrate VMs to other Ubuntu 22.04 hosts.
* Hosts running Ubuntu 24.04 can only migrate VMs to other Ubuntu 24.04 hosts.
* Cross-version migrations (Ubuntu 22.04 ↔ Ubuntu 24.04) are not supported.
**Known Limitation:** While initial migration from Ubuntu 22.04 to Ubuntu 24.04 may appear to succeed, subsequent migrations back to Ubuntu 22.04 may fail. This is due to incompatibilities between KVM versions on different operating systems.
#### Cross-OS Cold Migration: Ubuntu Hosts to Platform9 OS Hosts
Cold migration is supported from hosts running Ubuntu to hosts running **Platform9 OS powered by Rocky Linux by CIQ**. This is the supported path for transitioning your virtualized cluster from Ubuntu to Platform9 OS.
* Supported: Ubuntu 22.04 → Rocky Linux 10.2, Ubuntu 24.04 → Rocky Linux 10.2.
* The migration is **one-way**: cold migrating VMs back from Platform9 OS hosts to Ubuntu hosts is not supported.
* **Only cold migration is supported across operating systems.** Live migration and VM evacuation are still not supported. Power VMs off (or suspend them) before initiating the migration.
* All other Cold Migration Prerequisites still apply (CPU model compatibility, management network bandwidth, etc.).
#### Recommended Approach for OS Upgrades
When upgrading your cluster from Ubuntu 22.04 to Ubuntu 24.04 consider the following:
* Plan for one-way migration only (22.04 → 24.04).
* Complete all hosts upgrades before resuming normal migration operations.
* Do not attempt to migrate VMs back to older OS versions.
**When Migrating from Ubuntu Hosts to Platform9 OS Hosts**
When migrating from Ubuntu hosts to hosts running **Platform9 OS powered by Rocky Linux by CIQ**, consider the following:
* Cold-migrate VMs from Ubuntu source hosts to Platform9 OS destination hosts. Live migration and VM evacuation across OSes are not supported. Power VMs off (or suspend them) before initiating the migration.
* The migration is **one-way**. Do not attempt to cold-migrate VMs back to Ubuntu hosts after they have been moved to Platform9 OS hosts.
* Complete the migration of all VMs from a given Ubuntu host before decommissioning or repurposing it for Platform9 OS.
## Migration Limitations
### Cross-Cluster Live Migration
Cross-cluster live migration of virtual machines is not currently supported. This feature will be introduced with a future release of Private Cloud Director.
### Live Migration of GPU-Enabled VMs
You can live migrate virtual machines configured with [VGPUs](/private-cloud-director/gpu/gpu-support-pcd/setup-vgpu). VMs configured with GPU passthrough can not be live migrated today.
### Live Migration of Legacy Boot-from-Volume Hotplug VMs
VMs that use a hot-add-capable flavor and boot from a persistent volume, and that were created before the space.vars.product\_acronym 2025.10 release, may fail to live migrate until a one-time corrective step is applied. See [Resolve Live Migration Failures for Legacy Boot-from-Volume Hotplug VMs](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/hotplug-boot-from-volume-live-migration-failure).
# Flavorless VM Support
Although using flavors to create virtual machines is a fundamental feature of space.vars.product\_name, many organizations—especially those moving away from traditional virtualization vendors—may prefer a simpler process that doesn’t require specifying a flavor when creating a VM.
## Zero-Size Flavor
To support this scenario, Private Cloud Director allows creation of VMs using a **special zero-size flavor**, where both the **number of vCPUs and the RAM are set to zero**. Private Cloud Director ships out of the box with one such flavor, but you can also create new flavors where the number of vCPUs and the RAM are set to zero.
When you choose such a flavor when creating a new VM, you can then specify the amount of CPU and Memory that the VM should get provisioned with.
Using the zero-size flavor is a requirement in order to use CPU or memory hot-add feature
## Create a Zero-Size Flavor
1. In the Private Cloud Director UI, navigate to Virtual Machines ▷ Flavors and click Create Flavor.
2. **Enable Hot-plug**: toggle **on**.
3. Set a Disk size.
4. *(Optional)* Add Metadata if you want Private Cloud Director to place VMs on specific host aggregates.
5. Check the Make Public checkbox if you would like to make this flavor available to all tenants.
6. Click Create Flavor.
## Deploy a VM with a Zero-Size Flavor
1. Go to Virtual Machines ▷ Virtual Machines and click Deploy New VM.
2. During Flavor selection: enable the **Hot-add Compatible** toggle to filter the list.
3. Choose the zero-size flavor you created above.
4. In the Hotplug RAM (MB) and Hotplug vCPUs fields, enter the initial resources you would like to create this VM with. Maximum values are automatically configured by the system.
5. Finish the remaining steps and deploy the VM.
Private Cloud Director will create the VM with the requested CPU and Memory resources.
## Hot-add CPU or Memory to a Running VM
Read [VM Hot-Add CPU Or Memory](https://platform9.com/docs/private-cloud-director/2025.8/private-cloud-director/cpu-memory-hot-add-support) about how to hot add resources to a running VM.
## Use Flavorless VM with CLI
To automate creation of flavorless VM using CLI, API or terraform provider, use the zero-size flavor when creating the VM and use the following values for the metadata fields.
{% tabs %}
{% tab title="Bash" %}
```bash
{
"HOTPLUG_MEMORY": "",
"HOTPLUG_CPU": ""
}
```
{% endtab %}
{% endtabs %}
## Limitations & Best Practices
* VM disk size **cannot** be changed via the hot-add feature.
* On Windows Server 2022 guests, the memory value reported by the guest OS after a hot-add may not reflect the VM's current allocation. See [Limitations](/private-cloud-director/virtualized-clusters/virtualmachine/vm-hot-add-cpu-or-memory#limitations).
# VM Hot Add CPU Or Memory
space.vars.product\_name supports hot-add of CPU and / or memory resources to a running virtual machine. Using this feature, you can increase CPU cores or memory allocated to a running VM, without having to power-cycle it. This is valuable for production VMs that need more resources to meet workload demands but that can not be powered off.
### Pre-requisites
* Read more here about [Zero-Size Flavor](/private-cloud-director/virtualized-clusters/virtualmachine/flavorless-vms-with-hot-plug#zero-size-flavor). Using **zero-size flavor** when creating your VM is a requirement for hot-add. Before you can hot-add CPU or memory resources to a VM, the VM must have been created using zero-size flavor.
### Hot-add CPU or Memory to a Running VM
1. Select a powered-on VM that was created with a zero-size flavor.
2. On the VM grid view, navigate to **▷** **Other** **Actions ▷ Hotplug**.
3. Enter the new vCPU and/or memory value for the VM. The maximum allowable values are automatically set based on the VM configuration.
4. Click Hotplug VM.
### Limitations
* Hot add can only be used to increase the current CPU or memory allocation of a running VM. Reducing currently allocated CPU or memory is supported only while the VM is stopped.
* You can not use the hot-add feature to resize the disk of a running VM today.
* On VMs running Windows Server 2022, the guest OS's System Information (`systeminfo`) may display the flavor's maximum configured hotplug memory value instead of the VM's current memory allocation after a memory hot-add operation.
* Hot-add-capable VMs that boot from a persistent volume and were created before the space.vars.product\_acronym 2025.10 release may fail to live migrate until a one-time corrective hotplug is applied. See [Resolve Live Migration Failures for Legacy Boot-from-Volume Hotplug VMs](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/hotplug-boot-from-volume-live-migration-failure).
# Virtual Machine Snapshot
A virtual machine snapshot is an image of a virtual machine's disk at a specific point in time. It's a read-only copy that can be used to create a new virtual machine or restore a VM to a previous state. Essentially, it's a backup mechanism and a templating tool rolled into one.
You can create **two types** of VM snapshots in space.vars.product\_name.
## Type 1 - VM Ephemeral Disk Snapshot
This type of snapshot is created when you take a snapshot of a virtual machine that is using an ephemeral disk as it's root disk. A new image gets created in the Image Library that captures the point in time state of the VM's disk, including its data and configuration, at the moment the snapshot is taken.
**Snapshot File Type** **& Location**
* Ephemeral disk snapshots are always of type `qcow2`
* They are always stored in the Image Library Service as a single `qcow2` file.
Note that if a VM is created with ephemeral root disk, and then mounts one or more block storage volumes as data volumes, taking a snapshot of such a VM will only create a snapshot of it's root disk. The data volumes will not be snapshotted or tracked as part of the VM snapshot. To fully recreate such VM using a snapshot in the future, you need to first snapshot the VM and then separately take a volume snapshot of the data volumes. Then during VM restore or recreation time, you can create a new VM first using the VM snapshot, and then mount the volumes data volumes on this VM using the volume snapshots you created above.
{% hint style="danger" %}
**Important**
Taking a snapshot of a VM with ephemeral root disk will only snapshot the root disk. Any volumes attached to the VM will NOT be included in the snapshot.
{% endhint %}
### Live Snapshots
A snapshot is considered live when taken against a running virtual machine with no downtime.
Such snapshot is simply disk-only snapshot and as such is guaranteed to be crash consistent but *not application consistent.*
When you issue the command to create a snapshot of a running VM, the space.vars.product\_name compute service informs the hypervisor to freeze the VM to allow the creation of a “delta” file before resuming the execution of the VM. This is done to prevent the VM from writing directly to its disk while the disk is copied. VM continues to write to the delta file while the VM disk is being copied. When the copy is done, the VM is frozen again to allow the “delta” to be merged with the VM's disk, and the execution is then resumed with the disk fully merged.
Inconsistencies can appear on the first freeze if the VM is not aware that the hypervisor is taking a snapshot, because the applications and the kernel running on the instance are not told to flush their buffers.
For applications requiring strict data consistency, it is recommended to either shut down the VM or use application-level mechanisms to ensure data is flushed to disk before taking the snapshot.
{% hint style="warning" %}
**Warning**
Live VM snapshots are disk-only snapshots and may not capture the in-memory state of applications. For applications requiring strict data consistency, it is recommended to either shut down the VM or use application-level mechanisms to ensure data is flushed to disk before taking the snapshot.
{% endhint %}
## Type 2 - VM Volume Snapshot
This type of snapshot is created when you snapshot a virtual machine that is using a volume for it's root disk. When you snapshot such a virtual machine:
**Snapshot File Type and Location**
* The VM's root disk volume is snapshotted as a bootable volume snapshot.
* If the VM has any other volumes mounted, **those volumes are also snapshotted**.
* The Image Library service creates a **metadata entry** in the Image Library database that tracks a reference to the root volume snapshot and the data volume snapshots, along with other required information.
* **No other file gets created or stored** in the Image Library for this snapshot.
When a new VM gets created using this snapshot, the Image Library service will use the metadata stored in the database behind the scenes to identify the root volume snapshot and the data volume snapshots if any. This information will then be supplied to the Compute Service which will then use those snapshots to create a new VM.
## Snapshot Creation
You can take a snapshot of virtual machines in running or powered off state. In the UI, select the snapshot action for the VM to take the snapshot.
## Snapshot Properties
All VM snapshots will have the `image_type=snapshot` property set. You can view this property in the properties column.
## Delete a VM Snapshot
To delete a VM snapshot, navigate to **Images & VM Snapshots** > **VM Snapshots**, select one or more snapshots, and choose the delete action. The wizard opens a **Delete Snapshot(s)** dialog listing the snapshots being removed.
When you delete a VM Volume Snapshot, the dialog also offers a **Delete all the associated volume snapshots** checkbox. The checkbox is selected by default and removes the root and data volume snapshots that the VM snapshot references in the same operation:
> The associated volume snapshot(s) for the selected VM snapshot(s) will be deleted as part of this operation. Uncheck the checkbox if you wish to retain them.
The dialog lists the volume snapshots that will be deleted. Uncheck the box to keep those child volume snapshots after the parent VM snapshot is removed; in that case, only the VM snapshot's metadata entry is deleted.
# Virtual Machine Leasing
## VM Leasing
VM leasing lets you enforce automatic *power-off* or *deletion* of virtual machines (VMs) after a fixed amount of time. Leases are defined **once per tenant** and inherited by every existing and future VM that belongs to that tenant. Administrators and self-service users can also override the lease for an individual VM when needed.
### Lease-End Actions
| Action | Result | Typical Use Case |
| ------------- | ------------------------------------------------------------- | -------------------------------------------------- |
| **Power Off** | VM is gracefully shut down. Disks and metadata are retained. | Development or lab workloads you may revive later. |
| **Delete** | VM (and its attached ephemeral disks) is permanently removed. | Disposable environments or cost-control scenarios. |
## Tenant-Level Lease Policy
You can set the tenant-level lease policy by navigating to **Settings** **>** **Tenants & Users** **>** **Tenants** (on left navigation pane) **>** choose a **Tenant** **>** click the **Manage Lease Policy** button on the table header.
The tenant lease policy management screens allow you to manage the following lease configuration:
**Enable Lease Policy**: Turns leasing on or off for the tenant.
**Lease Duration**: The total time in *Days, Hours, Minutes* that each VM may run before the lease expires. The timer starts at VM creation.
**Action Upon Lease Expiration:** Defines the action triggered on lease expiration.
* **Power Off VM (default**): the VM is shut down.
* **Delete VM:** the VM is deleted.
{% hint style="info" %}
**Info**
Caution If you shorten the lease and existing VMs have already exceeded the new duration, the chosen lease-end action executes immediately.
{% endhint %}
## Per-VM Lease Customization
You can modify a lease at the VM level by navigating to **Virtual Machines** on the left-hand navigation pane **>** **Virtual Machines** **>** select a **virtual machine** **>** **Other** dropdown in the table header **>** **Customize Lease**
You can perform the following actions on the VM lease from the customization screen.
* **Specify action upon lease expiration:** Defines the action triggered on lease expiration for the selected VM. **This selection overrides the action set at the tenant level.**
* **Power Off VM (default**): the VM is shut down.
* **Delete VM:** the VM is deleted.
* **End sooner**: Pick a new date/time earlier than the current expiration to force an earlier shutdown or deletion.
* **Extend unexpired VMs**: Extend the lease by *up to one additional lease period* as defined in the tenant policy (e.g., +5 days if the policy is 5 days). You can repeat extensions indefinitely.
* **Extend expired VMs**: Even after a VM’s lease has expired, you can grant an extension (subject to the same maximum).
{% hint style="info" %}
**Info**
After a VM is powered-off by the lease engine, any manual power-on is only momentary. PCD will power the VM off again almost immediately.
{% endhint %}
## Permissions Matrix
The following table outlines the VM leasing permissions that the Admin, Self-Service, and Read-Only roles have.
| Role | Set Tenant Policy | Edit Lease (VM-level) | Create VM |
| --------------------- | ----------------- | --------------------- | --------- |
| **Admin** | ✔ | ✔ | ✔ |
| **Self-service User** | ✖ | ✔ | ✔ |
| **Read-only User** | ✖ | ✖ | ✖ |
## Troubleshooting
| Symptom | Cause | Resolution |
| -------------------------------------------- | ------------------------------------------ | ------------------------------------------------------ |
| VM powers off immediately after you start it | Lease already expired | Extend the lease or disable the tenant-level policy. |
| Cannot extend lease beyond a certain date | Max extension equals one full lease period | Increase the lease duration at the tenant level first. |
# Advanced Scheduling Options
This document describes advanced virtual machine scheduling strategies that you can use for specific enterprise use cases.
## NUMA Support
NUMA awareness allows the operating system of a virtual machine to intelligently schedule the workloads that it runs and minimize cross-node memory bandwidth.
In order to configure NUMA nodes for a virtual machine, you need to specify `hw:numa_nodes` property as part of a virtual machine flavor, then use that flavor to create the virtual machine.
For example, to restrict a virtual machine's vCPUs to a single host NUMA node, set `hw:numa_nodes=1`as part of the VM's flavor.
You can update an existing flavor and add this as new metadata - note that it will only apply to new VMs created using the flavor after the update. To update an existing flavor using `pcdctl` CLI, run the following command (replace with the name of the flavor you wish to update):
{% tabs %}
{% tab title="Bash" %}
```bash
pcdctl flavor set --property hw:numa_nodes=1
```
{% endtab %}
{% endtabs %}
Some workloads have very demanding requirements for memory access latency or bandwidth that exceed the memory bandwidth available from a single NUMA node. For such workloads, it is beneficial to spread the virtual machine across multiple host NUMA nodes, even if the VM's RAM/vCPUs could theoretically fit on a single NUMA node. To force a virtual machine's vCPUs to spread across two host NUMA nodes, set `hw:numa_nodes=2`as part of the VM's flavor.
The allocation of a virtual machine's vCPUs and memory from different host NUMA nodes can be configured. This allows for asymmetric allocation of vCPUs and memory, which can be important for some workloads. You can configure the allocation of VM vCPUs and memory across each VM NUMA node using the `hw:numa_cpus.{num}` and `hw:numa_mem.{num}` metadata properties as part of the VM flavor. For example, to spread the 6 vCPUs and 6 GB of memory of a VM across two NUMA nodes and create an asymmetric 1:2 vCPU and memory mapping between the two nodes, run the following command using `pcdctl` CLI (replace with the name of the flavor you wish to update):
{% tabs %}
{% tab title="Bash" %}
```bash
$ pcdctl flavor set --property hw:numa_nodes=2
$ pcdctl flavor set