Delete Orphaned Virtual Machine Entries

When a virtual machine is deleted / migrated / evacuated, sometimes its corresponding allocation on the source hypervisor host is not deleted from the PCD database.

Root Cause

• If the compute service on the source hypervisor host is stopped abruptly while a VM is being deleted / migrated / evacuated, then the records for the VM in the PCD management database may not be deleted fully.

• As a result, the host-side service cannot communicate with nova-conductor on the controller, so details are not shared and nova-compute may remain under the impression it still holds the virtual machine.

Resolution

Use the following steps to locate and remove orphaned allocations. Run the commands from a hypervisor host.

1

1. List stale allocations

Run nova-manage placement audit to list allocations for virtual machines that are either deleted or moved to other hypervisor hosts.

List stale allocations (run from a pod with nova-manage)

kubectl exec -it deploy/nova_api_osapi -n <NS> -- bash
nova-manage placement audit --verbose

Verify the output to identify suspected orphaned allocations.

2

2. Verify VM existence / location

Confirm whether the VM still exists or is hosted on another hypervisor host.

List servers on source host

pcdctl server list --all-projects --host <Source Host>

Show instance host

pcdctl server show -c OS-EXT-SRV-ATTR:host <instance-ID>

3

3. Identify the resource provider ID (if needed)

List resource providers to find the provider associated with the allocation.

List resource providers

pcdctl resource provider list

4

4. Delete a single orphaned allocation

After validating the allocation is stale and the instance is deleted or moved to another compute node, delete the orphaned allocation.

Delete specific allocation

kubectl exec -it <nova_api_osapi-pod_name> -n <NS> -- bash
nova-manage placement audit --verbose --delete <instance-ID>

5

5. Delete multiple/all orphaned allocations

If there are multiple orphaned allocations, delete them all at once.

Delete all orphaned allocations

nova-manage placement audit --verbose --delete

6

6. Heal allocations

After deletions, run heal_allocations to ensure Placement entries are consistent.

Heal allocations

nova-manage placement heal_allocations

7

7. Validate there are no more orphaned allocations

Re-run the audit to ensure no orphaned allocations remain.

Validate no orphaned allocations remain

nova-manage placement audit --verbose