> For the complete documentation index, see [llms.txt](https://docs.platform9.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.platform9.com/private-cloud-director/storage/troubleshooting-and-log-files/backend-selection-and-tuning.md). # Persistent Storage Service Backend Selection and Tuning ## Overview When you create a volume, the Persistent Storage Service selects a backend pool to host it. The selection is made by evaluating filters against all active pools and then applying weights to rank the passing pools. Understanding this process helps you diagnose unexpected placement outcomes — for example, a volume landing on the wrong backend, or a creation failing with `No valid host was found` when capacity appears available. In this guide, you will understand how backend selection works, inspect why a specific backend was or was not chosen, and tune placement behavior using volume types and extra specs. {% hint style="info" %} New to storage backends, pools, and the scheduler? Start with [Storage Scheduler and Storage Pools](/private-cloud-director/storage/block-storage/storage-scheduler-and-pools.md) for an introduction to these concepts, then return here to diagnose specific placement problems. {% endhint %} ## How the Persistent Storage Service Selects a Backend ### Step 1 — Gather Pool State Each block storage host periodically reports its pool statistics to the Persistent Storage Service's management plane. The report includes: * Total and free capacity for each pool (NFS share or backend pool). * Whether the pool supports thin provisioning, multiattach, and other capabilities. The management plane caches these statistics. By default, the cache is refreshed every 60 seconds. If a host goes unreachable, the cached values become stale; after a configurable timeout the pool is removed from consideration. ### Step 2 — Apply Filters The Persistent Storage Service applies a stack of filters. A pool must pass all active filters to be eligible. Key filters: | Filter | Purpose | | -------------------- | ------------------------------------------------------------------------------------------------------------- | | `CapacityFilter` | Pool must have enough free capacity for the requested volume size, after accounting for `reserved_percentage` | | `CapabilitiesFilter` | Pool must declare all capabilities listed in the volume type's extra specs | | `DriverFilter` | Allows per-backend custom expressions; rarely needed for standard deployments | ### Step 3 — Apply Weights Pools that pass all filters are ranked by weight. The default weight is based on free capacity (the pool with the most free capacity receives the highest weight and is selected first). This behavior spreads new volumes across pools to balance utilization. ### Step 4 — Select and Schedule The highest-weighted pool receives the volume. The block storage host managing that pool sends the provisioning command to the storage array. ## Inspect Why a Backend Was Chosen or Skipped ### List Active Pools and Their Capacity ```bash pcdctl --os-volume-api-version 3.12 volume backend pool list --detail ``` This command shows all pools the management plane knows about, along with their reported free and total capacity. Cross-reference this output against the `reserved_percentage` configured for each backend to understand how much capacity is actually available for placement. Available capacity after reservation: ``` effective_free = free_capacity_gb - (total_capacity_gb * reserved_percentage / 100) ``` ### Enable Scheduler Logging To see which pools pass or fail each filter, increase verbosity on the Persistent Storage Service scheduler. This is a temporary diagnostic step; revert after the investigation. {% hint style="info" %} **Self-Hosted deployments only** The following logging adjustment applies to the `cinder-scheduler` pod. In SaaS deployments, contact Platform9 support to enable scheduler debug logging for your region. {% endhint %} Add the following to the scheduler configuration and restart the `cinder-scheduler` pod: ```bash kubectl -n exec -it -- bash # Temporarily increase log level for diagnostic purposes only ``` In practice, the most useful diagnostic information is in the `cinder-scheduler` pod logs. Search for `filter_scheduler` and `No valid host`: ```bash kubectl logs -n | grep -E 'filter_scheduler|No valid host|AvailabilityZone|CapacityFilter' | tail -50 ``` The log output shows each filter's pass/fail decision per pool. A line like: ``` DEBUG cinder.scheduler.filters.capacity_filter ... Pool fails capacity check: free_capacity 5 < needed 10 ``` confirms that the pool was skipped because of insufficient free capacity. ### Check Whether Capacity Statistics Are Stale If a pool shows free capacity in `volume backend pool list` but volumes still fail to place on it, the statistics the scheduler is using may be stale. Force a refresh: ```bash # On the block storage host sudo systemctl restart pf9-cindervolume-base ``` After the service restarts, the host re-reports its pool statistics within 60 seconds. Retry the volume create. ## Common Misplacement Causes ### Volume Type Not Mapped to the Intended Backend A volume type contains extra specs that must match the destination pool's capabilities. If the extra specs do not match any pool, volume creation fails with `No valid host was found`. Verify the extra specs on your volume type: ```bash pcdctl volume type show ``` Check that the `volume_backend_name` extra spec matches the `volume_backend_name` value configured in your backend (`/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf`): ```bash sudo grep volume_backend_name /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf ``` ### `reserved_percentage` Set Too High A backend configured with a high `reserved_percentage` (for example, 30%) reserves that fraction of total capacity and does not offer it for placement. This can make a backend appear full even when the underlying storage array has available space. Check the current setting: ```bash sudo grep reserved_percentage /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf ``` Reduce the value if the reservation is larger than operationally required, then restart `pf9-cindervolume-base` for the new value to take effect. ### Multiple Pools on the Same Backend, One Exhausted When a backend is configured with multiple NFS shares or pools, the scheduler treats each as a separate candidate. If one pool is exhausted, new volumes should land on the remaining pools. However, if a volume type extra spec pins a `pool_name` pattern that matches only the exhausted pool, all creates fail. Review pool-level capacity: ```bash pcdctl --os-volume-api-version 3.12 volume backend pool list --detail ``` Look for pools where `free_capacity_gb` is near zero and `provisioned_capacity_gb` is close to `total_capacity_gb`. Either free space on that pool or update the extra spec to broaden the `pool_name` pattern. ## Influence Placement with Volume Types and Extra Specs ### Pin a Volume to a Specific Backend Set the `volume_backend_name` extra spec to the name of the target backend: ```bash pcdctl volume type set --property volume_backend_name= ``` All volumes created with this type will be placed on the named backend, regardless of capacity on other backends. ### Require a Specific Capability Use extra specs to require a capability that only certain backends support: ```bash # Require thin provisioning support pcdctl volume type set --property thin_provisioning_support=' True' # Require multiattach support pcdctl volume type set --property multiattach=' True' ``` Pools that do not advertise the required capability are filtered out. ### Spread Volumes Across Pools (Default Behavior) The default capacity-based weight already spreads volumes to the pool with the most free space. If you want to change weighting to prefer a specific pool, adjust weights using the `CapacityWeigher` multiplier in the backend configuration. Contact Platform9 support to apply custom scheduler weigher configuration for your region. ## Next Steps * Review [Volume Types and Backends](/private-cloud-director/storage/volume.md) for how to create volume types and associate them with backends. * For per-backend performance tuning, see [Per-Backend Performance Tuning and Troubleshooting](/private-cloud-director/storage/block-storage/volume-backend-configuration-examples/backend-performance-tuning.md). * For migration and retype placement failures, see [Volume Migration and Retype Troubleshooting](/private-cloud-director/storage/troubleshooting-and-log-files/volume-migration-retype-troubleshooting.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.platform9.com/private-cloud-director/storage/troubleshooting-and-log-files/backend-selection-and-tuning.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.