Volume Attach / Detach Troubleshooting

Overview

Volume attach and detach operations involve coordinated work between the Compute Service and the Persistent Storage Service. When either operation stalls or fails, a volume can become stuck in a state such as detaching, attaching, or reserved, blocking further operations on the volume and sometimes on the virtual machine.

This guide covers the most common root causes, how to detect each one, and the safe steps to recover a stuck volume without data loss.

In this guide, you will diagnose and recover volumes that are stuck in attach or detach states.

Prerequisites

• pcdctl configured and authenticated against your region.
• Access to the host running the affected VM (for systemctl and virsh commands).
• For Self-Hosted deployments: kubectl access to the management-plane namespace.

Common Causes

Symptom	Typical Root Cause
Volume stuck in detaching	Compute Service failed to complete the disconnect; volume record not cleaned up
Volume stuck in attaching	Transport-level failure (iSCSI/NFS) during the LUN-mapping step
Volume stuck in reserved	Compute Service crashed between reserving and completing the attachment
Stale attachment record	VM was deleted or migrated without detaching volumes first
Nova BDM inconsistency	Block device mapping record left over after a failed live migration
virsh commands hang; VM consoles inaccessible	A stale multipath device on the compute host left libvirtd unresponsive after a lost storage path
Image volume stuck in-use with a glance_store attachment	Image Library Service's temporary attachment for image read/write was not released

The last row above is specific to volumes that back an Image Library Service image rather than a VM. If the stuck volume's attachment device is glance_store and its server_id is empty, see Image Volume Stuck In-Use With a Stale glance_store Attachment for the recovery steps specific to that case.

Detect a Stuck Volume

Check Volume Status

pcdctl volume show <VOLUME_UUID>

Look at the status field. A volume in a healthy workflow should pass through intermediate states (attaching, detaching, reserved) in under two minutes. A volume that stays in one of those states for longer than five minutes is stuck.

To list all volumes in a potentially stuck state at once:

pcdctl volume list --all-projects | grep -E 'detaching|attaching|reserved|maintenance'

Identify the Attached VM and Host

pcdctl volume show <VOLUME_UUID> -f value -c attachments

The output includes the server_id (VM UUID) and the host_name of the compute node. Record both; you need them for subsequent steps.

Check the Attachment Record on the Compute Host

SSH to the compute host identified above and inspect the block device mapping:

sudo virsh domblklist <VM_UUID>

If the volume device appears in virsh output, the hypervisor still considers it attached. If virsh does not list it but the Persistent Storage Service shows in-use, the records are inconsistent.

Recover a Volume Stuck in detaching

A volume stuck in detaching most often means the Compute Service sent a detach request but did not receive confirmation from the storage host. Follow these steps in order.

Step 1: Verify the Compute Service Is Healthy

On the affected compute host:

sudo systemctl status pf9-ostackhost

If the service is not running, restart it and wait up to two minutes for in-flight operations to complete:

sudo systemctl restart pf9-ostackhost

Check whether the volume transitions to available on its own within two minutes:

watch -n 10 pcdctl volume show <VOLUME_UUID> -f value -c status

Step 2: Verify the Persistent Storage Service Is Healthy

On the block storage host:

sudo systemctl status pf9-cindervolume-base

Review logs for errors related to the volume UUID:

sudo grep <VOLUME_UUID> /var/log/pf9/cindervolume-base.log | tail -50

Self-Hosted deployments only

If the cinder-volume service runs as a pod rather than a systemd unit, check its status with:

kubectl get pods -n <WORKLOAD_REGION> | grep cinder-volume
kubectl logs -n <WORKLOAD_REGION> <CINDER_VOLUME_POD> | grep <VOLUME_UUID>

Step 3: Force-Reset the Volume State

If the Persistent Storage Service is healthy but the volume remains stuck, force-reset its state to available. This is safe only after you have confirmed that the volume is not genuinely connected to a running VM (see the virsh domblklist check above).

pcdctl volume set --state available <VOLUME_UUID>

Before resetting state

Confirm that virsh domblklist <VM_UUID> no longer lists the volume device. Resetting state on a volume that is still presented to a running VM can cause data corruption.

Step 4: Clean Up Stale Attachment Records

If a stale attachment record remains after the state reset, delete it:

# List attachment IDs for the volume
pcdctl volume attachment list --volume-id <VOLUME_UUID>

# Delete the stale attachment
pcdctl volume attachment delete <ATTACHMENT_UUID>

Step 5: Clean Up LUN Mappings on the Storage Backend

For SAN-backed volumes (iSCSI or Fibre Channel), the storage array may retain a LUN mapping even after the software-level detach completes. Check this from the storage management interface of your backend:

• NetApp ONTAP: verify that no igroup mapping exists for the host WWN / IQN that was using the volume.
• Pure Storage: verify that the host connection is removed in the Pure array management interface.
• Hitachi VSP: verify that the host group mapping no longer includes the LUN.

If a stale LUN mapping exists, remove it using your storage array's management tools. A stale mapping does not prevent the volume from being reused by a different VM, but it consumes resources on the storage array and can cause confusion during future attach operations.

Recover a Volume with Nova BDM Inconsistency

A Nova Block Device Mapping (BDM) inconsistency can occur when a live migration fails partway through and the migration rollback does not fully clean up. The volume appears in-use but is not attached to any running VM.

Detect the Inconsistency

# Show the VM the volume claims to be attached to
pcdctl volume show <VOLUME_UUID> -f value -c attachments

# Verify that VM exists and is in a stable state
pcdctl server show <VM_UUID>

If the VM does not exist, or is in ERROR state, the BDM record is stale.

Clean Up the Inconsistency

1. Reset the volume state to available:

   pcdctl volume set --state available <VOLUME_UUID>

2. Delete stale attachment records:

   pcdctl volume attachment list --volume-id <VOLUME_UUID>
   pcdctl volume attachment delete <ATTACHMENT_UUID>

3. If the VM is in ERROR state and you need to recover it, rebuild or delete it through the Compute Service. The volume will remain available and can be reattached to a new or recovered VM.

Recover from an Unresponsive libvirtd Due to Stale Multipath Maps

A multipath device that retains stale paths after a lost storage path (for example, after a volume is deleted) can leave libvirtd unresponsive on the compute host, causing virsh commands to hang and VM consoles to become inaccessible.

Detect the Issue

Check for a defunct QEMU process on the affected host:

ps aux | grep qemu | grep defunct

Check the multipath topology for a faulty or orphaned device:

multipath -ll

A device with all paths reported as failed or faulty, with no corresponding in-use volume, is a candidate for cleanup.

This is a known issue with no code fix planned. The steps below recover the host; they do not prevent recurrence on their own. See Prevent Recurrence below.

Remediation

1. Flush the stale multipath device map:

multipath -f <MULTIPATH_DEVICE>

1. If libvirtd remains unresponsive, restart it:

sudo systemctl restart libvirtd

Restarting libvirtd does not affect already-running VMs; it restarts only the management daemon that virsh and the Compute Service communicate with.

1. Verify that virsh responds normally:

sudo virsh list --all

Prevent Recurrence

Bound no_path_retry in /etc/multipath.conf on all compute and block-storage hosts so a lost path is retried a bounded number of times instead of leaving a stale device behind. See the multipath configuration guidance on the NetApp and Hitachi VSP storage-configuration pages.

Orphaned Attachments After Tenant or VM Deletion

When a tenant is deleted while volumes are still attached to VMs, or when VMs are force-deleted without detaching volumes first, orphaned attachment records can accumulate. These records cause subsequent volume operations to fail with "volume already in use" errors even though no active VM is using the volume.

To find orphaned attachments:

# List all attachments; cross-reference against running VMs
pcdctl volume attachment list --volume-id <VOLUME_UUID>
pcdctl server list --all-projects | grep <VM_UUID>

If the VM UUID from the attachment does not appear in the server list, the attachment is orphaned. Delete it:

pcdctl volume attachment delete <ATTACHMENT_UUID>
pcdctl volume set --state available <VOLUME_UUID>

Next Steps

• Review Volume State for a full list of volume status values and their meanings.
• If the volume was stuck because of a failed live migration, see Storage Live Migration for live-migration prerequisites and known limitations.
• For persistent or recurring attach/detach failures, see Troubleshooting Cinder Issues for service-level diagnostics.