Advanced Scheduling Options

This document describes advanced virtual machine scheduling strategies that you can use for specific enterprise use cases.

NUMA Support

NUMA awareness allows the operating system of a virtual machine to intelligently schedule the workloads that it runs and minimize cross-node memory bandwidth.

In order to configure NUMA nodes for a virtual machine, you need to specify hw:numa_nodes property as part of a virtual machine flavor, then use that flavor to create the virtual machine.

For example, to restrict a virtual machine's vCPUs to a single host NUMA node, set hw:numa_nodes=1as part of the VM's flavor.

You can update an existing flavor and add this as new metadata - note that it will only apply to new VMs created using the flavor after the update. To update an existing flavor using pcdctl CLI, run the following command (replace with the name of the flavor you wish to update):

Bash

pcdctl flavor set <flavor-name> --property hw:numa_nodes=1

Some workloads have very demanding requirements for memory access latency or bandwidth that exceed the memory bandwidth available from a single NUMA node. For such workloads, it is beneficial to spread the virtual machine across multiple host NUMA nodes, even if the VM's RAM/vCPUs could theoretically fit on a single NUMA node. To force a virtual machine's vCPUs to spread across two host NUMA nodes, set hw:numa_nodes=2as part of the VM's flavor.

The allocation of a virtual machine's vCPUs and memory from different host NUMA nodes can be configured. This allows for asymmetric allocation of vCPUs and memory, which can be important for some workloads. You can configure the allocation of VM vCPUs and memory across each VM NUMA node using the hw:numa_cpus.{num} and hw:numa_mem.{num} metadata properties as part of the VM flavor. For example, to spread the 6 vCPUs and 6 GB of memory of a VM across two NUMA nodes and create an asymmetric 1:2 vCPU and memory mapping between the two nodes, run the following command using pcdctl CLI (replace with the name of the flavor you wish to update):

Bash

$ pcdctl flavor set <flavor-name> --property hw:numa_nodes=2

$ pcdctl flavor set <flavor-name> \
  --property hw:numa_cpus.0=0,1 \
  --property hw:numa_mem.0=2048

$ pcdctl flavor set <flavor-name> \
  --property hw:numa_cpus.1=2,3,4,5 \
  --property hw:numa_mem.1=4096

Important

The {num} parameter is an index of VM NUMA nodes and may not correspond to host NUMA nodes. For example, on a platform with two NUMA nodes, the scheduler may opt to place VM NUMA node 0, as referenced in hw:numa_mem.0 on host NUMA node 1 and vice versa. Similarly, the CPUs bitmask specified in the value for hw:numa_cpus.{num} refer to VM vCPUs and may not correspond to host CPUs. As such, this feature cannot be used to constrain VMs to specific host CPUs or NUMA nodes.

Important

If the combined values of hw:numa_cpus.{num} or hw:numa_mem.{num} are greater than the available number of CPUs or memory respectively, this may result in VM provisioning failure.

CPU Pinning

By default, virtual machine vCPU processes are not assigned to any particular host CPU. This allows for features like overcommitting of CPUs. In heavily contended systems, this provides optimal system performance, however that may come at the expense of performance and latency for individual VMs.

Some virtualized workloads require real-time or near real-time behavior, which is not possible with the latency introduced by this default CPU scheduling policy. For such VMs, it is beneficial to control which host CPUs are bound to a VM's vCPUs. This process is known as CPU pinning. No other VMs can use the CPUs of a pinned VM, thus preventing resource contention between VMs.

You can configure a VM to use CPU pinning by specifying the hw:cpu_policy metadata property as part of the VM's flavor. There are three policies: dedicated, mixed and shared (the default). The dedicated CPU policy is used to specify that all CPUs of a VM should use pinned CPUs. To configure a flavor to use the dedicated CPU policy, run:

Bash

$ pcdctl flavor set <flavor-name> --property hw:cpu_policy=dedicated