> For the complete documentation index, see [llms.txt](https://docs.platform9.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.platform9.com/private-cloud-director/virtualized-clusters/virtualmachine/advance-vm-scheduling-options.md).

# Advanced Scheduling Options

This document describes advanced virtual machine scheduling strategies that you can use for specific enterprise use cases.

## NUMA Support

NUMA awareness allows the operating system of a virtual machine to intelligently schedule the workloads that it runs and minimize cross-node memory bandwidth.

In order to configure NUMA nodes for a virtual machine, you need to specify `hw:numa_nodes` property as part of a virtual machine flavor, then use that flavor to create the virtual machine.

For example, to restrict a virtual machine's vCPUs to a single host NUMA node, set `hw:numa_nodes=1`as part of the VM's flavor.

You can update an existing flavor and add this as new metadata - note that it will only apply to new VMs created using the flavor after the update. To update an existing flavor using `pcdctl` CLI, run the following command (replace with the name of the flavor you wish to update):

{% tabs %}
{% tab title="Bash" %}

```bash
pcdctl flavor set <flavor-name> --property hw:numa_nodes=1
```

{% endtab %}
{% endtabs %}

Some workloads have very demanding requirements for memory access latency or bandwidth that exceed the memory bandwidth available from a single NUMA node. For such workloads, it is beneficial to spread the virtual machine across multiple host NUMA nodes, even if the VM's RAM/vCPUs could theoretically fit on a single NUMA node. To force a virtual machine's vCPUs to spread across two host NUMA nodes, set `hw:numa_nodes=2`as part of the VM's flavor.

The allocation of a virtual machine's vCPUs and memory from different host NUMA nodes can be configured. This allows for asymmetric allocation of vCPUs and memory, which can be important for some workloads. You can configure the allocation of VM vCPUs and memory across each VM NUMA node using the `hw:numa_cpus.{num}` and `hw:numa_mem.{num}` metadata properties as part of the VM flavor. For example, to spread the 6 vCPUs and 6 GB of memory of a VM across two NUMA nodes and create an asymmetric 1:2 vCPU and memory mapping between the two nodes, run the following command using `pcdctl` CLI (replace with the name of the flavor you wish to update):

{% tabs %}
{% tab title="Bash" %}

```bash
$ pcdctl flavor set <flavor-name> --property hw:numa_nodes=2

$ pcdctl flavor set <flavor-name> \
  --property hw:numa_cpus.0=0,1 \
  --property hw:numa_mem.0=2048

$ pcdctl flavor set <flavor-name> \
  --property hw:numa_cpus.1=2,3,4,5 \
  --property hw:numa_mem.1=4096
```

{% endtab %}
{% endtabs %}

{% hint style="warning" %}
**Important**

The `{num}` parameter is an index of *VM* NUMA nodes and may not correspond to *host* NUMA nodes. For example, on a platform with two NUMA nodes, the Compute Service may opt to place VM NUMA node 0, as referenced in `hw:numa_mem.0` on host NUMA node 1 and vice versa. Similarly, the CPUs bitmask specified in the value for `hw:numa_cpus.{num}` refer to *VM* vCPUs and may not correspond to *host* CPUs. As such, this feature cannot be used to constrain VMs to specific host CPUs or NUMA nodes.
{% endhint %}

{% hint style="danger" %}
**Important**

If the combined values of `hw:numa_cpus.{num}` or `hw:numa_mem.{num}` are greater than the available number of CPUs or memory respectively, this may result in VM provisioning failure.
{% endhint %}

## NUMA Memory Allocation Policy

When you pin a virtual machine to a host NUMA node, the Compute Service also sets the memory allocation policy that the host applies to that VM's memory. The default policy is `strict`, which confines the VM's memory to the NUMA node it was placed on. If that node runs out of free memory, the host cannot fall back to another node, so it terminates the VM's process to reclaim memory. The VM shuts down even when other NUMA nodes on the same host still have free RAM.

The `hw:numa_mempolicy` flavor metadata property lets you relax that constraint. Set it to `preferred` so that a VM allocates from its assigned NUMA node while it can, and spills over to another node when that node is exhausted, instead of being terminated.

Set `hw:numa_mempolicy` to one of the following two values:

* `strict` (the default) confines the VM's memory to its assigned NUMA node. Choose this when a VM must never allocate memory from another node, accepting that the VM is shut down if its node is exhausted.
* `preferred` allocates from the assigned NUMA node first, then falls back to another node under memory pressure.

If you do not set `hw:numa_mempolicy`, the policy remains `strict` and VM behavior is unchanged.

To set the memory allocation policy from the UI, navigate to **Virtual Machines** in the left-hand navigation menu and open the **Flavors** page. Select the flavor you want to update, then choose **Edit Metadata** from the Actions menu. Under **Flavor Metadata**, use **Add Metadata** to add the key `hw:numa_mempolicy` with the value `preferred`, then click **Update Flavor**.

To make the same change using the `pcdctl` CLI, run the following command (replace with the name of the flavor you wish to update):

{% tabs %}
{% tab title="Bash" %}

```bash
$ pcdctl flavor set <flavor-name> \
  --property hw:numa_nodes=1 \
  --property hw:numa_mempolicy=preferred
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Applies to single-node VMs only**

The `preferred` policy applies only to a VM that occupies a single host NUMA node, because a host cannot apply a preferred policy across more than one node. If you combine `hw:numa_mempolicy=preferred` with `hw:numa_nodes=2` or higher, the Compute Service logs a warning and uses `strict` for that VM instead. The VM still starts normally.
{% endhint %}

### Apply the Policy to Existing VMs

Adding `hw:numa_mempolicy` to a flavor affects only the VMs you create from that flavor afterwards. The policy is recorded for a VM when the VM is created, so an existing VM keeps the policy it started with, and rebooting it does not pick up the new value from the flavor.

To move an existing VM onto the new policy, resize it to a flavor that sets `hw:numa_mempolicy`. The VM's memory allocation policy is regenerated as part of the resize.

Once a VM is running with the `preferred` policy, that policy persists across hard reboots, resizes, and live migrations.

## CPU Pinning

By default, virtual machine vCPU processes are not assigned to any particular host CPU. This allows for features like overcommitting of CPUs. In heavily contended systems, this provides optimal system performance, however that may come at the expense of performance and latency for individual VMs.

Some virtualized workloads require real-time or near real-time behavior, which is not possible with the latency introduced by this default CPU scheduling policy. For such VMs, it is beneficial to control which host CPUs are bound to a VM's vCPUs. This process is known as CPU pinning. No other VMs can use the CPUs of a pinned VM, thus preventing resource contention between VMs.

You can configure a VM to use CPU pinning by specifying the [`hw:cpu_policy`](https://docs.openstack.org/nova/latest/configuration/extra-specs.html#hw:cpu_policy) metadata property as part of the VM's flavor. There are three policies: `dedicated`, `mixed` and `shared` (the default). The `dedicated` CPU policy is used to specify that all CPUs of a VM should use pinned CPUs. To configure a flavor to use the `dedicated` CPU policy, run:

{% tabs %}
{% tab title="Bash" %}

```bash
$ pcdctl flavor set <flavor-name> --property hw:cpu_policy=dedicated
```

{% endtab %}
{% endtabs %}

## Related Pages

* [VM Flavors](/private-cloud-director/virtualized-clusters/virtualmachine/vm-flavors.md): how flavors and flavor metadata define a VM's resource allocation
* [Performance Tuning](/private-cloud-director/virtualized-clusters/performance-tuning.md): image properties that complement flavor-level tuning
* [Diagnose VM Scheduling Failures](/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/diagnose-vm-scheduling-failures.md): what to check when a NUMA-constrained VM fails to schedule


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.platform9.com/private-cloud-director/virtualized-clusters/virtualmachine/advance-vm-scheduling-options.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.