Image Library High Availability
You can create a highly available deployment of Private Cloud Director image library service in a region by enabling image library role on multiple hosts. This feature requires the use of shared storage backend for image library storage. Once configured, this feature enables seamless access to images across host failures or maintenance events for hosts assigned with image library role.
Prerequisites
Following are the pre-requisites for enabling high availability for image library service:
Networking and firewall rules allow image-related traffic between image library hosts and all other hosts configured with rest of the
services.
A shared storage backend is required. A shared storage backend (e.g., NFS or another supported block storage volume backend) is a mandatory prerequisite for enabling high availability (HA) of the Image Library Service. Assigning the Image Library role to multiple hosts without using a shared storage backend is not supported may lead to issues such as inconsistent image discovery, image deletion failures, or orphaned image data.
Shared storage must be accessible by all Image Library hosts. The shared backend must be mounted and available to all hosts that will be assigned the Image Library role with the same path.
All Image Library hosts must be in the same region. Cross-region image library deployments are not supported. All hosts assigned the Image Library role must reside within the same region.
Network connectivity must allow image-related traffic. Firewall and network policies must permit image-related traffic between all Image Library hosts and all other hosts configured with rest of the Private Cloud Director services.
Supported Storage Backends
Following table describes the supported and unsupported backends for configuring high availability for image library service.
Backend Type
HA Support
File-based (e.g., NFS)
Supported, but it must be mounted on every image library host at the same mount point/path. This gives consistency between image library hosts to prevent corruption, deletion mismatches, or discovery issues.
Block Storage Volume
Recommended for production because it offers better scalability, reliability and image transfer performance compared to file-based storage. Still needs to be accessible by all image library hosts.
How High Availability Works in Image Library
When images are stored on shared storage, any image library host can serve them for VM or volume creation.
During image creation, Private Cloud Director dynamically selects an available and healthy image library host. If one of the image library hosts is offline, the system transparently retries with another active image library host to create the image, ensuring uninterrupted service.
Deployment Steps
Step 1: Configure Shared Storage Backend
The image library can be configured to use block storage as it's backend. You can do this by providing the name of the volume type for the block storage backend you'd like to use while specifying image library location as part of the cluster blueprint configuration. This informs the image library service to use block storage as the persistent backend to save and retrieve images.
Alternatively, if using file-based shared storage (e.g., NFS), it must be mounted or attached on each host where the image library role will be enabled with the exact same path**.**
mount -t nfs <NFS_SERVER>:_<EXPORTED_PATH> _var_opt_imagelibrary_data
ls -l _var_opt_imagelibrary_dataStep 2: Enable Image Library Role on Multiple Hosts
Navigate to Infrastructure > Hosts in the UI.
Select the target host.
Click Edit Roles and assign the Image Library role.
Repeat this for all hosts that should be part of your highly available image library service setup.
Step 3: Validate that image library service is running
Run the following command on each host enabled with the image library role:
systemctl status pf9-glance-apiCheck that:
The pf9-glance-api service is active.
No errors are reported in
/var/log/pf9/glance-api.log.
You can also validate from the UI by checking the Settings > API Access > API Endpoints and check that image-cluster service is available with multiple image library endpoints.
Image Library Admin Endpoint
Read more about Image Library Admin Endpoint here. In case of highly available image library setup with multiple hosts having image library role assigned, the last host to get the image library service role assigned is selected to be the admin endpoint.
The admin endpoint is primarily used to upload images to the image library. When creating a new virtual machine, the compute and the block storage service are configured to round robin across all available image library hosts to fetch the required image.
If an image library host that is also acting as an admin endpoint goes down, the admin endpoint is not automatically assigned to one of the other surviving image library hosts today. You will need to manually change the admin endpoint to a different image library host (by following the steps below).
Note that this limitation only impacts your ability to upload new images to the image library. It does not impact new virtual machine provisioning. The admin endpoint is only used to upload new images to the image library. When creating a new VM, the compute and block storage service are designed to use any of the available image library hosts in a round robin fashion the fetch the virtual machine image.
Important
If an image library host that is also acting as an admin endpoint goes down, the admin endpoint is not automatically assigned to one of the other surviving image library hosts today. You will need to manually change the admin endpoint to a different image library host (by following the steps below). This is required so you can continue to upload images to the image library service.
You can manually configure or change the image library admin endpoint assignment by running the following pcdctl command:
Step 1 - Get the admin endpoint UUID
Run the following command to get UUID of the admin endpoint. This command will list the ID of the current admin endpoint. Copy it.
pcdctl endpoint list --service glance --interface adminExample output:
pcdctl endpoint list --service glance --interface admin
+----------------------------------+--------+--------------+--------------+---------+-----------+----------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+--------+--------------+--------------+---------+-----------+----------------------------+
| 4bf27ff9f8a146d59dcce04bcedb7mz0 | SJC | glance | image | True | admin | https:__111.11.33.138:9494 |
+----------------------------------+--------+--------------+--------------+---------+-----------+----------------------------+Step 2 - Set the Admin Endpoint
Run the following command to set the new admin endpoint. Replace with the ip address of your alternate image library host. Replace with the ID that you copied from the command above.
pcdctl endpoint set --url https:__<IP>:9494 <UUID>Using our previous example, and say the IP address of your second image service host is 111.11.33.139, then the following command will set this host to be the image library service admin endpoint:
pcdctl endpoint set --url https:__111.11.33.139:9494 4bf27ff9f8a146d59dcce04bcedb7mz0Last updated
Was this helpful?