Volume Encryption with an External Key Manager

Overview

Private Cloud Director supports encrypting Persistent Storage Service volumes, including boot volumes. By default, encryption keys are created and managed by the platform's built-in key manager, which runs in the management plane.

Starting with this release, you can configure the Persistent Storage Service and the Compute Service to use HashiCorp Vault as an external key manager backend instead. With an external key manager, encryption keys are created directly on the dataplane and retrieved from your Vault instance, rather than from the platform's built-in key manager. This is useful for environments that require encryption keys, including those used for boot volumes, to stay outside the platform's management plane.

In this guide, you will configure PCD hosts to use HashiCorp Vault as the key manager backend for volume encryption.

Configure this before any encrypted volumes exist

Switching the key manager backend on a host that already has encrypted volumes makes those volumes' existing keys unreachable. The volume data remains on disk but can no longer be decrypted. Configure Vault as the key manager backend before you create any encrypted volumes.

Prerequisites

• Root privileges on each host you configure.
• A running HashiCorp Vault instance, reachable from every host with the Persistent Storage Node role and every hypervisor host that will run VMs using encrypted volumes.
• A Vault token that's restricted to only the storage location used for these encryption keys, not a token with full access to your Vault instance.

The Vault token and its associated configuration details come from your organization's Vault administrator. Before starting Step 1, request the following from them:

• A token restricted to a single storage location, created using a policy like this one (they'll adjust the path to match their own Vault layout):
• The following details, which you'll enter in Step 1:

  • The mount name used for the path above (Vault's default is secret).
  • The path assigned above, if any.
  • Whether their Vault instance uses KV version 2 (the current default) or the older KV version 1.
  • If a token isn't practical under your organization's security policy, whether they can set you up with AppRole, JWT, or Kubernetes-based authentication instead.

How It Works

When you configure an external key manager backend:

• The Persistent Storage Service dataplane component, running on each host with the Persistent Storage Node role, creates and stores encryption keys directly in the configured Vault instance during volume creation, instead of requesting them from the platform's built-in key manager.
• The Compute Service retrieves the same keys directly from Vault when attaching an encrypted volume to a VM, or booting a VM from an encrypted boot volume.
• Because both services must resolve the same keys, the Persistent Storage Service and the Compute Service on every host must point at the same Vault instance.

Both services must point to the same Vault instance

If the Persistent Storage Service and the Compute Service are configured against different Vault instances, volume creation or VM boot from an encrypted volume will fail: the service consuming the key won't be able to find it.

Configure Vault as the Key Manager Backend

Step 1 - Configure the Persistent Storage Service

On each host with the Persistent Storage Node role, open the block storage override configuration file:

Bash

sudo vi /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf

Add the following configuration:

Bash

[key_manager]
backend = vault

[vault]
vault_url = http://<vault-host>:<port>
root_token_id = <vault-token>
# kv_mountpoint = <mount-path>   # only if the KV engine is not mounted at the default path "secret"
# kv_path = <secrets-path>       # only if your Vault deployment uses a non-default secrets path
# kv_version = 2                 # defaults to 2 (KV version 2); set to 1 only if your Vault uses KV version 1

root_token_id is not Vault's cluster root token

Despite its name, set root_token_id to a Vault token generated from a policy scoped only to the secrets path used for volume encryption keys, not Vault's actual cluster root token. A token with unrestricted Vault access would give the Persistent Storage Service and the Compute Service far more access than volume encryption needs.

If root-token authentication doesn't fit your environment, the Vault backend also supports AppRole, JWT, and Kubernetes-based authentication using the auth_method, approle_role_id, approle_secret_id, token_role, token_file, and auth_path options.

Save the file, then restart the block storage service for the change to take effect:

Bash

sudo systemctl restart pf9-cindervolume-base

Step 2 - Configure the Compute Service

The Compute Service needs the same key manager configuration to retrieve encryption keys when it attaches an encrypted volume, or boots a VM from an encrypted boot volume. Follow Configure an External Key Manager for Volume Encryption on every hypervisor host that will run VMs using encrypted volumes.

Notes

• vault_url must be reachable from the host you're configuring.
• The configured Vault token must have access to the secrets path used to store encryption keys, matching the policy described in Prerequisites.
• root_token_id doesn't renew automatically. If it expires or is revoked, update it with a new token from your Vault administrator and restart the affected service.
• Configure both the Persistent Storage Service and the Compute Service to point at the same Vault instance: a key created during volume creation must be resolvable during instance boot or volume attach.
• Restart both the block storage service and the Compute Service after applying these changes.
• Only HashiCorp Vault is supported as an external key manager backend at this time.

Next Steps

See Compute Service Advanced Configuration and Advance Service Configuration for other host-level configuration options.