Backup and Restore Management Plane

This guide provides steps for backing up and restoring the self-hosted Private Cloud Director management plane in disaster recovery scenarios. The procedures include both manual and automated backup methods, as well as manual restoration process.

Info

When restoring the management plane, ensure it's done on a Kubernetes cluster that is separate from the cluster where the backup was generated.

Prerequisites

System Requirements

• Access to the Kubernetes management cluster

• Installed and configured airctl binary

• Valid airctl configuration file at /opt/pf9/airctl/conf/airctl-config.yaml

• Root or sudo access to the management node

For S3 Backup Storage

• AWS credentials with S3 bucket access

• Existing S3 bucket for backup storage

• AWS CLI configured (for verification purposes)

Important Considerations

1. The restoration process must be performed on a separate Kubernetes management cluster that is different from the management cluster where the backup was generated.

2. The metrics service (gnocci) data is not backed up in the backup procedure. For complete disaster recovery, manual copying of metrics service (gnocchi) metrics from the original storage class pcd-sc persistent volume is required.

Manual Backup Procedure

Create a backup directory:

Bash

mkdir -p /root/backup-mgmt/

Execute the airctl backup command:

Bash

airctl backup --outdir /root/backup-mgmt/ --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose

Verify backup contents:

Bash

tar tvf /root/backup-mgmt/backup.tar.gz

The backup archive should contain:

• state_backup.yaml: System state configuration

• kplane_values_backup.yaml: Kubernetes management cluster configuration

• consul.snap: Consul snapshot

• mysql_dump_Infra.sql: Infrastructure database backup

• mysql_dump_Region1.sql: Region-specific database backup

• ovn-north-backup & ovn-south-backup : Ovn database backup

Warning

Metrics service (gnocci) persistent data will not be backed up or restored as part of the above procedure. Therefore, in a full disaster recovery scenario, you must manually copy the metrics service (gnocchi) metrics data from the original storage class pcd-sc persistent volume.

Automated Backup Configuration

The automated backup system is created during the initial installation of the Private Cloud Director management plane. When you run the installation command, the system automatically creates a CronJob named mgmt-plane-backup. This cronjob is designed to run every hour to ensure regular system backups.

Backups are stored at path /var/pf9/backups/ on the node with airctl installation

Bash

$ kubectl get cronjob 
NAME                SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
mgmt-plane-backup   0 * * * *   False     0        2m56s           12m

Configuring S3 Backup Storage

Locate the aws-credentials secret, Empty secret aws-credentials created by default::

Bash

kubectl get secret aws-credentials -o yaml | grep AWS

Configure the secret with required AWS parameters:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

• AWS_REGION

• AWS_S3_PATH

Info

If the secret is patched with the required AWS credentials and S3 bucket information, the cronjob execution will store backups locally and push them to the S3 bucket.

Once configured, backups will be stored both locally and in the specified S3 bucket.

Manual Restore Procedure

Standard Restore

Execute the restore command:

Bash

airctl restore --backupdir /root/backup-mgmt --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose

Restore from S3 Backup

1. Create and configure the aws-credentials secret with required AWS parameters

2. Execute the S3 restore command

Bash

airctl restore --s3backup --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose

Info

For complete disaster recovery, manually restore Gnocchi metrics data from the original pcd-sc persistent volume

Verification Steps

Check backup file integrity using MD5 checksum::

Bash

# Generate MD5 checksum for the backup file
md5sum <backup-file>.tar.gz

# Optional: Compare with a pre-recorded checksum
# You can save the MD5 checksum when initially creating the backup
md5sum /root/backup-mgmt/backup.tar.gz > backup-checksum.txt

# Later, verify the backup file matches the original checksum
md5sum -c backup-checksum.txt

Verify S3 uploads (if configured):

Bash

aws s3 ls s3://<bucket-name>/<backup-path>

Monitor restore progress:

Bash

kubectl logs -f <restore-pod-name>

Common Issues

• If AWS credentials are not properly configured, automated S3 backups will continue locally but skip S3 upload

• Restore operations may take significant time depending on data volume

• Services may take additional time to start after restore completion