Persistent Storage Service Backend Selection and Tuning

Overview

When you create a volume, the Persistent Storage Service selects a backend pool to host it. The selection is made by evaluating filters against all active pools and then applying weights to rank the passing pools. Understanding this process helps you diagnose unexpected placement outcomes — for example, a volume landing on the wrong backend, or a creation failing with No valid host was found when capacity appears available.

In this guide, you will understand how backend selection works, inspect why a specific backend was or was not chosen, and tune placement behavior using volume types and extra specs.

New to storage backends, pools, and the scheduler? Start with Storage Scheduler and Storage Pools for an introduction to these concepts, then return here to diagnose specific placement problems.

How the Persistent Storage Service Selects a Backend

Step 1 — Gather Pool State

Each block storage host periodically reports its pool statistics to the Persistent Storage Service's management plane. The report includes:

• Total and free capacity for each pool (NFS share or backend pool).

• Whether the pool supports thin provisioning, multiattach, and other capabilities.

The management plane caches these statistics. By default, the cache is refreshed every 60 seconds. If a host goes unreachable, the cached values become stale; after a configurable timeout the pool is removed from consideration.

Step 2 — Apply Filters

The Persistent Storage Service applies a stack of filters. A pool must pass all active filters to be eligible. Key filters:

Filter	Purpose
CapacityFilter	Pool must have enough free capacity for the requested volume size, after accounting for reserved_percentage
CapabilitiesFilter	Pool must declare all capabilities listed in the volume type's extra specs
DriverFilter	Allows per-backend custom expressions; rarely needed for standard deployments

Step 3 — Apply Weights

Pools that pass all filters are ranked by weight. The default weight is based on free capacity (the pool with the most free capacity receives the highest weight and is selected first). This behavior spreads new volumes across pools to balance utilization.

Step 4 — Select and Schedule

The highest-weighted pool receives the volume. The block storage host managing that pool sends the provisioning command to the storage array.

Inspect Why a Backend Was Chosen or Skipped

List Active Pools and Their Capacity

pcdctl --os-volume-api-version 3.12 volume backend pool list --detail

This command shows all pools the management plane knows about, along with their reported free and total capacity. Cross-reference this output against the reserved_percentage configured for each backend to understand how much capacity is actually available for placement.

Available capacity after reservation:

effective_free = free_capacity_gb - (total_capacity_gb * reserved_percentage / 100)

Enable Scheduler Logging

To see which pools pass or fail each filter, increase verbosity on the Persistent Storage Service scheduler. This is a temporary diagnostic step; revert after the investigation.

Self-Hosted deployments only

The following logging adjustment applies to the cinder-scheduler pod. In SaaS deployments, contact Platform9 support to enable scheduler debug logging for your region.

Add the following to the scheduler configuration and restart the cinder-scheduler pod:

kubectl -n <WORKLOAD_REGION> exec -it <CINDER_SCHEDULER_POD> -- bash
# Temporarily increase log level for diagnostic purposes only

In practice, the most useful diagnostic information is in the cinder-scheduler pod logs. Search for filter_scheduler and No valid host:

kubectl logs -n <WORKLOAD_REGION> <CINDER_SCHEDULER_POD> | grep -E 'filter_scheduler|No valid host|AvailabilityZone|CapacityFilter' | tail -50

The log output shows each filter's pass/fail decision per pool. A line like:

DEBUG cinder.scheduler.filters.capacity_filter ... Pool <NAME> fails capacity check: free_capacity 5 < needed 10

confirms that the pool was skipped because of insufficient free capacity.

Check Whether Capacity Statistics Are Stale

If a pool shows free capacity in volume backend pool list but volumes still fail to place on it, the statistics the scheduler is using may be stale. Force a refresh:

# On the block storage host
sudo systemctl restart pf9-cindervolume-base

After the service restarts, the host re-reports its pool statistics within 60 seconds. Retry the volume create.

Common Misplacement Causes

Volume Type Not Mapped to the Intended Backend

A volume type contains extra specs that must match the destination pool's capabilities. If the extra specs do not match any pool, volume creation fails with No valid host was found.

Verify the extra specs on your volume type:

pcdctl volume type show <VOLUME_TYPE_NAME>

Check that the volume_backend_name extra spec matches the volume_backend_name value configured in your backend (/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf):

sudo grep volume_backend_name /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf

reserved_percentage Set Too High

A backend configured with a high reserved_percentage (for example, 30%) reserves that fraction of total capacity and does not offer it for placement. This can make a backend appear full even when the underlying storage array has available space.

Check the current setting:

sudo grep reserved_percentage /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf

Reduce the value if the reservation is larger than operationally required, then restart pf9-cindervolume-base for the new value to take effect.

Multiple Pools on the Same Backend, One Exhausted

When a backend is configured with multiple NFS shares or pools, the scheduler treats each as a separate candidate. If one pool is exhausted, new volumes should land on the remaining pools. However, if a volume type extra spec pins a pool_name pattern that matches only the exhausted pool, all creates fail.

Review pool-level capacity:

pcdctl --os-volume-api-version 3.12 volume backend pool list --detail

Look for pools where free_capacity_gb is near zero and provisioned_capacity_gb is close to total_capacity_gb. Either free space on that pool or update the extra spec to broaden the pool_name pattern.

Influence Placement with Volume Types and Extra Specs

Pin a Volume to a Specific Backend

Set the volume_backend_name extra spec to the name of the target backend:

pcdctl volume type set --property volume_backend_name=<BACKEND_NAME> <VOLUME_TYPE_NAME>

All volumes created with this type will be placed on the named backend, regardless of capacity on other backends.

Require a Specific Capability

Use extra specs to require a capability that only certain backends support:

# Require thin provisioning support
pcdctl volume type set --property thin_provisioning_support='<is> True' <VOLUME_TYPE_NAME>

# Require multiattach support
pcdctl volume type set --property multiattach='<is> True' <VOLUME_TYPE_NAME>

Pools that do not advertise the required capability are filtered out.

Spread Volumes Across Pools (Default Behavior)

The default capacity-based weight already spreads volumes to the pool with the most free space. If you want to change weighting to prefer a specific pool, adjust weights using the CapacityWeigher multiplier in the backend configuration. Contact Platform9 support to apply custom scheduler weigher configuration for your region.

Next Steps

• Review Volume Types and Backends for how to create volume types and associate them with backends.

• For per-backend performance tuning, see Per-Backend Performance Tuning and Troubleshooting.

• For migration and retype placement failures, see Volume Migration and Retype Troubleshooting.