Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Image Service Troubleshooting Guide

Problem

A troubleshooting guide for image services is needed to address frequent issues with image management in cloud environments, such as image upload failures, slow performance, and incorrect metadata. The guide must provide clear, actionable steps for diagnosing and resolving common errors to ensure the reliability and availability of the image service.

Environment

  • Private Cloud Director Virtualization - v2025.4 and Higher

  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

  • Component - PCD Image service

Deep Dive

Image Creation Flow

The image creation process in space.vars.product_name is managed by the Glance service. The flow begins when a user uploads a new image file, which is then processed and stored.

1

User Request

A user initiates an image upload via the OpenStack CLI, space.vars.product_name dashboard, or direct API call. The request includes the image file and metadata (e.g., name, format, disk format).

2

API Service

The Glance API service receives the request, validates the user's authentication token with Keystone, and checks for permissions and quotas. Below Glance API logs show the token is being used.

INFO glance.api.v2.image_data [None [REQ-ID] [USER_ID] [TENANT_ID] - - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.
3

Image Validation and Upload Request

The glance service validates the image format and confirms that its virtual size (can be fetched using the command qemu-img info <image_name>.qcow2 on glance host) meets the requirements. Then the Image PUT /v2/images/<IMAGE_UUID>/file request is placed to upload image data to a temporary staging area, ideally at the default staging location (If default directory changed, then check the custom location) /var/lib/glance/os_glance_staging_store/.

INFO glance.location [None [REQ-ID] [USER_ID] [TENANT_ID] - - default default] Image format matched and virtual size computed: 41126400
INFO eventlet.wsgi.server [None [REQ-ID] [USER_ID] [TENANT_ID] - - default default] 127.0.0.1 - - [..] "PUT /v2/images/[IMAGE_UUID]/file HTTP/1.0" 204 468 2.400140
4

Glance API to Registry

The API service then communicates with the Glance Registry, which creates a new entry for the image in the Glance database. The status is set to queued.

5

Glance Service

The Glance API hands off the request to the pf9-glance-api service, which moves the image data from the staging area to the backend storage (e.g., Swift, Ceph, or a local file system) default image file storage location /var/opt/imagelibrary/data/glance/.

6

Status Update

Once the image is successfully stored, the Glance Store service updates the image's status in the database from queued to active. The image is now ready for use. The host glance audit logs (/var/log/pf9/glance-audit.log) show information about the request Username, Image UUID, outcome, etc.

INFO oslo.messaging.notification.audit.http.response [None [REQ-ID] [USER_ID] [TENANT_ID] - - default default] {"message_id": "[Audit_Message_ID]", "publisher_id": "glance-api", "event_type": "audit.http.response", "priority": "INFO", "payload": {"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "eventType": "activity", "id": "[Activity_ID]", "eventTime": "[..]", "action": "update", "outcome": "success", "observer": {"id": "target"}, "initiator": {"id": "[..]", "typeURI": "service/security/account/user", "name": "[USER_NAME]", "credential": {"token": "***", "identity_status": "Confirmed"}, "host": {"address": "127.0.0.1", "agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"}, "project_id": "[PROJECT_UUID]"}, "target": {"id": "unknown", "typeURI": "unknown", "name": "unknown"}, "requestPath": "/v2/images/[IMAGE_UUID]/file", "tags": ["correlation_id?value=[..]"], "reason": {"reasonType": "HTTP", "reasonCode": "204"}, "reporterchain": [{"role": "modifier", "reporterTime": "[..]", "reporter": {"id": "target"}}]}, "timestamp": "[..]"}

Image Deletion Flow

The deletion process also uses the Glance services to remove the image's data and its database entry.

1

User Request

A user sends a deletion request via the OpenStack CLI, space.vars.product_name dashboard, or direct API call. The request includes the image information (e.g Image ID or Name).

2

API Service

The Glance API service receives the request, validates the user's authentication token with Keystone, and performs a permission check, and changes the image's status in the database to deleting. Below Glance API logs show the token is being used.

3

Glance Service

The pf9-glance-api service receives a message to delete the image from the backend storage. The Glance API hands off the request to the pf9-glance-api service, which moves the image data from backend storage (e.g., Swift, Ceph, or a local file system) default image file storage location /var/opt/imagelibrary/data/glance/. This is a crucial step that frees up disk space.

4

Final Status Update

Once the data is confirmed to be deleted from the backend store, the Glance API removes the image's database entry, completing the deletion process.

Procedure

The following steps outline how to troubleshoot the image issue.

1

Review image details

Review image details like status and any errors.

2

Validate Glance endpoints

Validate if the glance image endpoints are available and the public endpoint is responding using a curl request. This curl request should return the glance information.

3

Check if the image service is enabled

4

Check glance-api pod (Self-Hosted only)

The management plane has a glance-api pod to provide the image service. Check if the glance-api pod is running in the workload region namespace. Review this pod:

Info

Step 4 is applicable only for Self-Hosted Private Cloud Director

  • Check if they are in "CrashLoopBackOff/OOMkilled/Pending/Error/Init" state.

  • Also, verify if all containers in the pods are Running.

  • See the events section in pod describe output.

  • Review pods logs using REQ_ID or VM_UUID for relevant details.

5

Validate pf9-glance-api service on host

Validate if the pf9-glance-api service is running on the host where glance role is applied.

6

Review host logs

On the host, review the /var/log/pf9/glance-api.log to track the relevant events against a specific image ID.

7

Escalation

If these steps prove insufficient to resolve the issue, kindly reach out to the Platform9 Support Team for additional assistance.

Most Common Causes

  • Ensure that the Image Library Service Pre-Requisites are met.

  • While uploading an image, the admin.rc file does not have the OS_INTERFACE variable set to admin.

  • Incorrect image format. See Supported Image Formats.

  • The pf9-glance-api service is down on the Image Library host.

  • In Self-Hosted deployments, the --insecure flag was not used with the pcdctl command and the Image Library host uses a self-signed certificate.

  • Uploading an image to a volume whose type is both encrypted and backed by an NFS storage backend: see Diagnose Image Upload Failures on Encrypted NFS-Backed Volumes.

  • A volume-based VM creation fails because the source image's Persistent Storage Service volume is stuck in-use with a stale glance_store attachment: see Image Volume Stuck In-Use With a Stale glance_store Attachment.

Troubleshoot Image Upload and Queued Status

Overview

When you upload an image, it passes through the following status sequence: queued → saving → active. An image that stays in queued indefinitely means the Image Library Service accepted the metadata but was unable to receive or store the file data. The sections below walk through the most common causes and how to resolve them.

Check Disk Space on the Image Library Host

The Image Library Service writes image data to /var/opt/imagelibrary/data/glance/ by default. If the filesystem holding that path is full, the upload stalls in queued and the service logs an I/O or disk-full error.

  1. On the Image Library host, check available disk space:

  1. If the filesystem is at or near 100%, free space by removing unused images or expanding the filesystem before retrying the upload.

  2. Check the Image Library Service log for disk-related errors:

Verify the Image Library Service Is Running

A stopped or crashed pf9-glance-api service will cause all uploads to stall in queued.

If the service is not active, start it and then check for errors that might have caused it to stop:

Verify Image Format

An unsupported or mismatched image format can prevent the service from processing the upload. Confirm the actual format of the image file before uploading:

Use the format reported by qemu-img info when specifying --disk-format in the pcdctl command or selecting the format in the UI. Supported formats are raw, qcow2, and iso.

Check for Network or Timeout Issues

Large image uploads over slow or intermittent network connections can time out before the data fully transfers.

  • Upload from a machine that has a direct, low-latency network path to the Image Library host.

  • Check for network errors in the Image Library Service log:

Recover a Stuck Image

If an image is stuck in queued and you have verified that none of the above conditions apply, delete the stuck image record and re-upload:

Then retry the upload. If the upload consistently stalls on re-upload, contact Platform9 Support with the image UUID and the relevant entries from /var/log/pf9/glance-api.log.

Differences Between UI and CLI Upload Failures

Upload Path
What to Check

UI upload button is inactive

Image Library Service certificate is not trusted in your browser: see Image Library Service Certificate Configuration.

UI upload starts but stalls

Check disk space and service health as described above.

CLI upload returns SSL: CERTIFICATE_VERIFY_FAILED

Add --insecure to the pcdctl image create command.

CLI upload returns Connection refused or Unable to establish connection

The Image Library endpoint is not reachable: see Image Library Service Endpoint Health.

Image Library Service Endpoint Health

Overview

When the Image Library Service endpoint is unreachable, image uploads fail and VM provisioning that requires fetching an image from the Image Library host will also fail. The most common symptoms are:

  • pcdctl image create returns a connection error or HTTP 502 / 503.

  • A curl request to the Image Library endpoint returns an NGINX error page (502 Bad Gateway, 503 Service Unavailable).

  • The Service Health dashboard in the UI shows the Image Library Service as unhealthy.

Check the Image Library Service Endpoints

List the registered Image Library Service endpoints and verify that they are enabled:

An endpoint with Enabled: False will not serve requests. If any endpoint shows as disabled, re-enable it or reassign the Image Library role to the host.

Test the Endpoint Directly

Use curl to check whether the Image Library endpoint responds:

  • A JSON response that includes version information indicates the endpoint is healthy.

  • An NGINX error page (HTML with "502 Bad Gateway" or "503 Service Unavailable") means the NGINX reverse proxy on the Image Library host is running but the upstream pf9-glance-api process is not accepting connections.

Check the Service on the Image Library Host

When the endpoint returns an NGINX error, the pf9-glance-api service on the Image Library host has likely stopped or is not listening:

Restart both services if either is not active:

After restarting, check the log for startup errors:

Review Logs

The primary log for Image Library Service issues is on the Image Library host:

For audit events (upload outcomes, user identity, image UUID):

Self-Hosted deployments only

In Self-Hosted deployments, the management-plane glance-api pod in the region namespace provides the endpoint that the Image Library host proxies through. If the host-level service appears healthy but the endpoint still returns errors, check the management-plane pod:

In SaaS deployments, the management-plane components are operated by Platform9. If endpoint errors persist after verifying host-level service health, contact Platform9 Support.

Escalation

If service restarts do not resolve the issue and the endpoint continues to return errors, contact Platform9 Support with:

  • Output of pcdctl endpoint list --service glance

  • Output of sudo systemctl status pf9-glance-api

  • The last 200 lines of /var/log/pf9/glance-api.log

Persistent Storage Backend and Image Caching at Scale

Overview

The Image Library Service supports two storage backends: a local file store and a Persistent Storage Service (block storage) volume. When the block storage backend is configured, image data is stored on a block storage volume rather than on the Image Library host's local filesystem. This is the recommended configuration for production environments and for Image Library High Availability.

For background on configuring the block storage backend and enabling Image Library High Availability, see Image Library High Availability and Block Storage High Availability.

How Hypervisor Image Caching Works

When a virtual machine is provisioned for the first time using a given image, the hypervisor (compute host) must fetch the image data from the Image Library host over the network. After the first successful boot from that image on a given hypervisor, the hypervisor retains a local copy of the image in its image cache. Subsequent VM deployments using the same image on the same hypervisor use the cached copy and avoid the full image transfer.

This caching behavior has important implications for bulk and parallel VM deployments:

  • First deployment of an image on a hypervisor is the most I/O-intensive. If many VMs are deployed simultaneously using an image that has never been cached on those hypervisors, all of them will attempt to fetch the full image from the Image Library host at the same time.

  • Image Library host bandwidth is the bottleneck. The Image Library host (or hosts, in an HA setup) must serve the full image to every hypervisor that does not yet have a cached copy. With many hypervisors requesting simultaneously, this can saturate the Image Library host's network interface and cause VM provisioning to time out or fail.

  • Block storage backend improves throughput. When the Image Library Service uses a block storage volume as its backend, the image data is fetched from the storage array rather than from the Image Library host's local disk, which can improve parallelism for large deployments.

Recommendations for Bulk Deployments

When you need to deploy many VMs simultaneously (for example, during a large-scale workload rollout), consider the following:

  • Pre-warm the cache. Before the bulk deployment, deploy a single VM on each hypervisor using the target image. This populates the cache on all hypervisors. Subsequent bulk deployments will use the cached copy and avoid the simultaneous image-fetch bottleneck.

  • Use Image Library HA with shared storage. An Image Library deployment with multiple hosts sharing a block storage backend distributes read requests across hosts, reducing the load on any single host during the initial image fetch. See Image Library High Availability.

  • Stage bulk deploys in waves. If pre-warming is not practical, deploy VMs in smaller batches rather than all at once, allowing each batch to complete its image fetch before the next batch starts.

Troubleshoot Slow or Failing Bulk Deploys

If VM provisioning fails or times out during a large bulk deployment and image fetching is suspected:

  1. Check the Image Library host's network utilization during the deployment window. Sustained high network I/O concurrent with provisioning failures points to an image-fetch bottleneck.

  2. Check /var/log/pf9/glance-api.log on the Image Library host for errors or timeouts corresponding to the failed provisioning requests.

  3. Verify that the Image Library Service endpoint is healthy before the deployment: see Image Library Service Endpoint Health.

  4. If the Image Library uses a block storage backend, verify that the storage volume is healthy and that the Persistent Storage Service is operating normally.

If issues persist, contact Platform9 Support with the Image Library host logs and a description of the deployment scale (number of VMs, number of hypervisors, image size).

Image Volume Stuck In-Use With a Stale glance_store Attachment

Overview

When the Image Library Service uses a Persistent Storage Service volume as the backend for an image, it temporarily attaches that volume using the device name glance_store while it reads or writes image data. After the operation completes, the Image Library Service detaches the volume and its status returns to available.

If the detach request does not complete, for example because the Persistent Storage Service API is temporarily unreachable or times out, the temporary attachment is never cleaned up. The volume remains in in-use state with a stale glance_store attachment even though it is not attached to any virtual machine. Because the image volume stays locked in this state, subsequent volume-based VM creation requests that need to read the image fail.

Symptoms

  • Creating a volume-based VM from the affected image fails, while pcdctl volume show on the image's backing volume reports status: in-use.

  • The volume's attachment list shows a device named glance_store with no server_id, meaning the attachment is not associated with any running VM.

  • The Persistent Storage Service log shows a connection or timeout error for an attachment-delete request around the time of the failed image operation.

Diagnose

  1. Identify the volume backing the affected image and inspect its status and attachments:

  1. Confirm the in-use volume has a glance_store attachment with an empty server_id. This confirms the volume is not genuinely attached to a VM.

  2. Check the Persistent Storage Service log for attachment-delete errors around the time of the failure:

Resolve

  1. Delete the stale attachment:

  1. If the volume status does not return to available after the attachment is removed, reset it manually:

  1. Retry the volume-based VM creation.

This issue is typically caused by transient connectivity problems between the Image Library Service and the Persistent Storage Service. If it recurs frequently, check the network path and load between the two services.

Related Pages

PreviousImage Creation Failed Using CLI
NextCopy Image Fails Between NFS and Tintri Backends

Last updated

Was this helpful?