Troubleshooting GPU Support

When you configure GPU support or deploy GPU-enabled VMs, you might encounter configuration errors or issues with GPU resource availability. The following troubleshooting information addresses specific problems that have been identified during GPU setup and operations.

NOTE

Before troubleshooting GPU issues, verify that your GPU model is supported and that you have completed all infrastructure setup steps. Most GPU errors result from incomplete configuration or mismatched settings between hosts and flavors.

vGPU script fails with SR-IOV unbindLock error

When running the vGPU configuration script, you may encounter this error during SR-IOV configuration.

Bash

Step 3: Enable SR-IOV for NVIDIA GPUs
Detecting PCI devices from /sys/bus/pci/devices...
Found the following NVIDIA PCI devices:
Found the following NVIDIA devices: 0000:c1:00.0
Enter the full PCI device IDs (e.g., 0000:17:00.0 0000:18:00.0) to enable sriov, separated by spaces.
Press Enter without input to configure ALL listed NVIDIA GPUs:
No PCI device IDs provided. Configuring all NVIDIA GPUs...
Enabling SR-IOV for 0000:c1:00.0...
Enabling VFs on 0000:c1:00.0
Cannot obtain unbindLock for 0000:c1:00.0

This error occurs when NVIDIA services are holding a lock on the PCI device, preventing SR-IOV configuration.

Prerequisites for this resolution

• NVIDIA license and drivers are installed successfully

• NVIDIA license server is created and configured

Resolution steps:

1. Stop all NVIDIA-related services that might be holding the device lock:

Bash

systemctl stop nvidia-vgpu-mgr
systemctl stop nvidia-persistenced || echo "nvidia-persistenced not running"
systemctl stop nvidia-dcgm || echo "nvidia-dcgm not running"
killall -q nv-hostengine || echo "nv-hostengine not running"

1. Remove and rescan the PCI device to reset its state:

Bash

echo 1 > /sys/bus/pci/devices/0000:c1:00.0/remove 
echo 1 > /sys/bus/pci/rescan

Replace 0000:c1:00.0 with your actual PCI device ID from the error message.

1. Manually enable SR-IOV for the GPU device:

Bash

/usr/lib/nvidia/sriov-manage -e 0000:c1:00.0

1. Restart the NVIDIA vGPU manager:

Bash

systemctl start nvidia-vgpu-mgr

1. Verify SR-IOV configuration by checking for virtual functions:

Bash

nvidia-smi vgpu 
mdevctl list

Expected output after successful resolution:

Bash

# nvidia-smi vgpu output
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.06    Driver Version: 570.148.06                       |
|---------------------------------+------------------------------+------------|
| GPU  Name                       | Bus-Id                       | GPU-Util   |
| vGPU ID  Name                   | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  NVIDIA L4                  | 00000000:C1:00.0             |      0%    |
| 3251634321  NVIDIA L4-1A        | ef2a...   ef2acb59-4400-4... |      0%    |
+---------------------------------+------------------------------+------------+

# mdevctl list output
8bfe3867-7918-4da6-b6f9-1a3ff4db030c 0000:c1:02.6 nvidia-918

vGPU VMs fail to recover after GPU host reboot

When a GPU host with vGPU VMs is rebooted, the VMs scheduled on that host do not automatically recover. The system fails to recreate the required mediated device (mdev) configurations after the host restarts.

This occurs because mdev devices are not persistent across reboots. When the host restarts, the SR-IOV configuration and mdev device mappings are lost, preventing vGPU VMs from accessing their assigned GPU resources.

Resolution steps:

1. Before rebooting the host, capture the current mdev device configuration:

Bash

root@elaborate-rate:~ mdevctl list

1. Save this output.

You will need the device UUIDs and PCI addresses. Your output will look similar to the following:

Bash

295ba0f1-20d3-4a94-9d24-ad29edc5ac15 0000:c1:01.1 nvidia-924
916542a1-f966-4195-97d6-7cf2a10c850d 0000:c1:03.2 nvidia-924

1. After the host reboots, reconfigure SR-IOV by running the GPU configuration script:

Bash

./pf9-gpu-configure.sh

1. Select option 3 when prompted:

Bash

Select configuration type:
1) PCI Passthrough
2) vGPU pre configure
3) vGPU SR-IOV configure
4) Validate Passthrough
5) Validate vGPU
Enter choice (1 - 5): 3
...
...

1. Recreate the mdev devices for each vGPU VM using the UUIDs and PCI addresses you saved in step 1:

Bash

echo 916542a1-f966-4195-97d6-7cf2a10c850d > /sys/class/mdev_bus/0000:c1:03.2/mdev_supported_types/nvidia-924/create
echo 295ba0f1-20d3-4a94-9d24-ad29edc5ac15 > /sys/class/mdev_bus/0000:c1:01.1/mdev_supported_types/nvidia-924/create

Replace the UUIDs, PCI addresses, and nvidia-924 profile names with the values from your saved output.

1. Verify the mdev devices are recreated:

Bash

mdevctl list

The output should match your saved configuration from step 1.

Your vGPU VMs should now be able to start and access the assigned GPU resources.

vGPU profiles missing during host configuration

When configuring vGPU on a GPU host, certain vGPU profiles may be missing from the available options, even though your GPU model supports them.

This occurs when existing mediated devices (stale or active) are present in /sys/bus/mdev/devices/. The system prevents selection of vGPU profiles that would create fewer or equal slices than the number of existing devices.

For example, if 2 devices are present at /sys/bus/mdev/devices/, any vGPU profile that allows 2 or fewer GPU slices will not be listed under GPU hosts for configuration.

Resolution steps:

1. Check for existing mediated devices:

Bash

ls /sys/bus/mdev/devices/

1. Count the number of devices present:

Bash

ls /sys/bus/mdev/devices/ | wc -l

1. Stop NVIDIA vGPU services:

Bash

systemctl stop nvidia-vgpu-mgr

1. Remove existing mediated devices:

Bash

for device in /sys/bus/mdev/devices/*; do
    if [ -d "$device" ]; then
        echo 1 > "$device/remove"
    fi
done

1. Restart NVIDIA vGPU services:

Bash

systemctl start nvidia-vgpu-mgr

1. Verify the devices are cleared:

Bash

ls /sys/bus/mdev/devices/

1. Return to the GPU host configuration in PCD to see the full list of available vGPU profiles.

VM creation fails with GPU model validation error

If you select a GPU model in the VM flavor that doesn't match the configured GPU on the host, the system will provide an error during VM creation. This ensures that the selected GPU in the flavor aligns with the underlying GPU hardware.

To resolve this issue, verify that the GPU model specified in your flavor matches the GPU model configured on the host.

GPU host authorization and visibility issues

GPU host does not appear as a listed GPU host

After running the GPU configuration script and authorizing, the GPU host should appear in the GPU host list showing compatibility (passthrough or vGPU), GPU model, and device ID.

If your GPU host does not appear:

• Verify the GPU configuration script ran successfully on the host.

• Confirm you rebooted the host after running the script.

• Check that host authorization completed.

• Ensure both host configuration and cluster have GPU enabled.

Script execution requirements

The GPU configuration script requires administrator privileges to run. Only administrators with access to the host and script should execute it. End users or developers requesting GPU resources do not need to run the script themselves.

Incorrect host configuration resulting in Cold Migration failure and Flavor resize of GPU enabled VMs

1. Failure of GPU enabled VM Cold Migration.

Error in Ostackhost logs post cold migration attempt:

Ostackhost log

Insufficient compute resources: Claim pci failed.

1. Failure of Flavor resize of GPU enabled VMs

Resizing the flavor of VMs are failing for Flavor upgrade and Flavor downgrade with error:

UI Error

Invalid PCI alias definition: Device type mismatch for alias 'nvidia-14'

Steps to resolve the issues related to the incorrect host configurations:

1. Deauthorise existing GPU enabled hosts part of the host configuration that needs to be corrected/modified. Reference: Remove all roles and deauthorize a Host

Affected Host

$ pcdctl deauthorize-node

1. Run the below GPU passthrough script to cleanly remove the existing GPU configurations to avoid any conflicts in future:

Affected Host

#!/bin/bash

echo "### Reverting GPU PCI passthrough configuration... ###"

# Remove VFIO modprobe config
if [ -f /etc/modprobe.d/vfio.conf ]; then
    echo "Removing /etc/modprobe.d/vfio.conf"
    rm -f /etc/modprobe.d/vfio.conf
fi

# Remove blacklist files
if [ -f /etc/modprobe.d/blacklist-nvidia.conf ]; then
    echo "Removing /etc/modprobe.d/blacklist-nvidia.conf"
    rm -f /etc/modprobe.d/blacklist-nvidia.conf
fi

if [ -f /etc/modprobe.d/blacklist-nouveau.conf ]; then
    echo "Removing /etc/modprobe.d/blacklist-nouveau.conf"
    rm -f /etc/modprobe.d/blacklist-nouveau.conf
fi

# Remove KVM config
if [ -f /etc/modprobe.d/kvm.conf ]; then
    echo "Removing /etc/modprobe.d/kvm.conf"
    rm -f /etc/modprobe.d/kvm.conf
fi

# Clean initramfs-tools/modules
if [ -f /etc/initramfs-tools/modules ]; then
    echo "Cleaning VFIO modules from /etc/initramfs-tools/modules"
    sed -i '/^vfio$/d' /etc/initramfs-tools/modules
    sed -i '/^vfio_pci$/d' /etc/initramfs-tools/modules
    sed -i '/^vfio_iommu_type1$/d' /etc/initramfs-tools/modules
fi

echo "Updating initramfs..."
update-initramfs -u

# Restore GRUB config
GRUB_FILE="/etc/default/grub"
if [ -f "$GRUB_FILE" ]; then
    echo "Cleaning GRUB_CMDLINE_LINUX_DEFAULT in $GRUB_FILE"
    sed -i 's/vfio-pci\.ids=[^ ]*//g' "$GRUB_FILE"
    sed -i 's/vfio_iommu_type1\.allow_unsafe_interrupts=1//g' "$GRUB_FILE"
    sed -i 's/modprobe\.blacklist=[^ ]*//g' "$GRUB_FILE"
    sed -i 's/kvm\.ignore_msrs=1//g' "$GRUB_FILE"
    sed -i 's/  / /g' "$GRUB_FILE"
fi

echo "Updating GRUB..."
update-grub

# Unbind GPU from vfio-pci
echo "Attempting to unbind NVIDIA GPUs from vfio-pci and rebind to nvidia driver..."
for PCI in $(lspci -nn | grep -i nvidia | awk '{print $1}'); do
    echo "Processing $PCI"
    if [ -e /sys/bus/pci/devices/0000:$PCI/driver ]; then
        DRIVER=$(basename "$(readlink /sys/bus/pci/devices/0000:$PCI/driver)")
        if [ "$DRIVER" = "vfio-pci" ]; then
            echo "Unbinding $PCI from vfio-pci"
            echo 0000:$PCI > /sys/bus/pci/devices/0000:$PCI/driver/unbind
            echo "Rebinding $PCI to nvidia driver"
            echo 0000:$PCI > /sys/bus/pci/drivers/nvidia/bind 2>/dev/null || echo "Could not bind $PCI to nvidia driver now (may require reboot)"
        fi
    fi
done

echo "### Revert complete. Please reboot the system to fully apply changes. ###"

1. Now decommission the GPU host. Reference: Decommission a Host

Affected Host

$ pcdctl decommission-node

1. Add the required GPU configurations in the Host config section in the Cluster blueprint from the Management Plane UI. Authorize your GPU Hosts

2. Now onboard the GPU host to the Management plane. Reference: Authorize Host

3. From the GPU host run the gpu-passthrough script /opt/pf9/gpu/pf9-gpu-configure.sh to configure the GPU settings within the GPU host. Reference: Run the GPU configuration script

Bash

sudo ./opt/pf9/gpu/pf9-gpu-configure.sh

1. Re-authorise the GPU host with the correct host configuration: Reference: Authorize your GPU Hosts