Maintenance Mode

Private Cloud Director maintenance mode enables administrators to perform routine maintenance operations on their fleet of hosts in the data center in a safe and orchestrated manner. Maintenance mode allows you to safely migrate virtual machines from a host and prevent new virtual machines from being scheduled during maintenance operations.

What Happens in Maintenance Mode?

When maintenance mode is enabled for a host:

• All running VMs on the host are automatically live-migrated to another compute host in the cluster.

  • You have the option to choose the host to which the VMs will get migrated

• The host is marked as unschedulable, ensuring no new VMs are placed on it.

• Once all VM migrations are complete, the host remains online but isolated from scheduling operations, allowing administrators to perform software updates, hardware maintenance, or diagnostics.

Enabling Maintenance Mode

To enable maintenance mode on a host:

1. Navigate to Infrastructure → Cluster Hosts in the Private Cloud Director UI.

2. Select the target host from the list (optional). If no target host is selected, maintenance mode will select an appropriate target host from the cluster.

3. Click Other → Enable Maintenance Mode.

As noted in point 2 above, you may choose to:

• Allow Private Cloud Director to automatically determine the best destination for each VM based on cluster capacity and placement policies.

• Manually specify a destination host for all VM migrations.

Click Migrate VMs and Enable Maintenance Mode to initiate the process.

After host maintenance operations are completed, you can disable maintenance mode to allow the host to rejoin the scheduler.

Disabling Maintenance Mode

Before disabling maintenance mode, ensure that any maintenance tasks on the host have been completed successfully and that the host is healthy and ready to handle new workloads.

1. Navigate to Infrastructure → Cluster Hosts.

2. Locate the host that is currently in Maintenance Mode.

   1. Hosts in maintenance mode are typically marked with a "Maintenance Mode" label or status indicator.

3. Select the host, then click on Other → Disable Maintenance Mode

The host will be marked as schedulable again and Private Cloud Director will resume placing new virtual machines on this host.

UI Observability

The Private Cloud Director UI provides real-time status indicators and detailed migration progress information throughout the maintenance mode lifecycle.

Maintenance Mode Status on the Cluster Hosts Page

The Infrastructure → Cluster Hosts page reflects the current maintenance mode state for each host:

• Scheduling column: When maintenance mode is enabled on a host, the Scheduling column displays Disabled to indicate that the host is no longer accepting new VM placements.

• Maintenance Mode label: During the transition into maintenance mode, a status label displays Entering Maintenance Mode in the Scheduling column. This label is visible in the hosts table for any host that is currently undergoing the maintenance mode process.

• Other Actions menu: When a host is in maintenance mode, the Enable Maintenance Mode option in the Other Actions dropdown is disabled, and the Disable Maintenance Mode option becomes available. The Enable/Disable Scheduling option is also disabled while maintenance mode is active, since scheduling is managed by the maintenance mode process.

Maintenance Mode Status on the Host Details Page

Clicking on a host that is in maintenance mode opens the Host details page, which displays a banner at the top of the page with the following information:

• Migration status summary: The banner indicates whether VM migrations are in progress or have completed successfully.

• VM count: The total number of VMs that were migrated out of the total that will be migrated.

• Timestamps: The start time of the maintenance mode operation is displayed. The completion time is displayed once all migrations are finished.

• See Details: A button that opens the View Migration Progress panel with detailed per-VM migration status.

• Disable Maintenance Mode: A button that allows you to disable maintenance mode directly from the Host details page without navigating back to the hosts list.

View Migration Progress

Clicking See Details on the Host details page banner opens the View Migration Progress panel. This panel provides a detailed breakdown of the maintenance mode operation:

• Overall Migration Status: Displays the current state of the migration operation (for example, Completed, In Progress, or Failed).

• Number of VMs Migrated: Shows the count of VMs that have been successfully migrated out of the total.

• Start Time: The timestamp when the maintenance mode operation was initiated.

• Completion Time: The timestamp when all migrations finished (displayed upon completion).

• Virtual Machines Migration table: Lists each individual VM that was migrated, along with its migration status. This allows administrators to identify any VMs that may have failed to migrate and retry migration.

Maintenance Mode Interoperation with Other Services

This section describes how maintenance mode interoperates with other services configured for your cluster.

Host Aggregates

Maintenance mode will honor Host Aggregates and migrate VMs to another host from the same host aggregate.

Limitations:

• All hosts from the Host Aggregate must belong to a single cluster and not span multiple clusters.

• If the host aggregate has a single host that has gone down, maintenance mode will not find a suitable target to migrate the VMs, and maintenance mode will fail.

DRR

A host in maintenance mode will have scheduling VMs disabled while maintenance mode is active. Consequently, DRR will exclude any hosts that are in maintenance mode in a given cluster, and only balance workloads on the remaining active hosts.

VMs with Hard Affinity or Anti-Affinity Rules

• Hard affinity: Maintenance mode will fail if one or more VMs with hard affinity rules are in a powered-on state.

• Soft affinity: Maintenance mode will attempt to keep all VMs in the affinity group together by migrating the first VM to a randomly selected host in the cluster and then migrating the subsequent VMs to the same host until it reaches capacity.

• Hard anti-affinity: Maintenance mode will honor the anti-affinity policy by placing the VMs in the anti-affinity group on separate hosts. If no appropriate target host is found to migrate a VM, maintenance mode will fail.

• Soft anti-affinity: Maintenance mode will attempt to find target hosts that satisfy the anti-affinity policy for each VM to be migrated in a powered-on state, ensuring that all VMs part of the soft anti-affinity policy are placed on separate hosts. If maintenance mode cannot find a suitable target host, maintenance mode will migrate the VM(s) to a target host that violates the soft affinity policy, and the policy violation will be displayed on the UI.

VM States

Maintenance mode will migrate VMs that are in an active state (powered on).

Maintenance mode will not migrate VMs with the following statuses:

• Shutdown (powered off)

• Suspended

• Error

• Unknown

Maintenance mode will not proceed if a VM exists in the following states:

• Pending resize confirmation: The resize operation needs to be confirmed for the VM to be eligible for migration by maintenance mode.

• Rescued: The VM must be un-rescued for the VM to be eligible for migration by maintenance mode.

VMs with Special Properties

1. Virtual TPM-enabled VMs: Maintenance mode does not have the ability to live migrate VMs with Virtual TPM enabled today. This ability is coming soon.

2. Zero flavor VMs: Maintenance mode will migrate VMs created with zero flavor in a powered-on state.

3. VMs with hot-added CPU or memory: Maintenance mode will migrate VMs that have hot-added CPU or memory resources.