# Backup and Restore Management Plane

This guide provides steps for backing up and restoring the self-hosted <code class="expression">space.vars.product\_name</code> management plane in disaster recovery scenarios. The procedures include both manual and automated backup methods, as well as manual restoration process.

{% hint style="info" %}
**Info**

When restoring the management plane, ensure it's done on a Kubernetes cluster that is separate from the cluster where the backup was generated.
{% endhint %}

## Prerequisites

#### System Requirements

* Access to the Kubernetes management cluster
* Installed and configured `airctl` binary
* Valid `airctl` configuration file at `/opt/pf9/airctl/conf/airctl-config.yaml`
* Root or sudo access to the management node

#### For S3 Backup Storage

* AWS credentials with S3 bucket access
* Existing S3 bucket for backup storage
* AWS CLI configured (for verification purposes)

## Important Considerations

1. The restoration process must be performed on a separate Kubernetes management cluster that is different from the management cluster where the backup was generated.

Manual Backup Procedure

1. Create a backup directory:

{% tabs %}
{% tab title="Bash" %}

```bash
mkdir -p /tmp/backup-mgmt/
```

{% endtab %}
{% endtabs %}

2. Execute the airctl backup command:

{% hint style="info" %}
**Info**

Execute the following commands as a non-root user.
{% endhint %}

{% tabs %}
{% tab title="Bash" %}

```bash
airctl backup --outdir /tmp/backup-mgmt/ --config /opt/pf9/airctl/conf/airctl-config.yaml  --verbose
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

Use `--region <region_name>` parameter if you intend to back up only a specific region. If not specified, all the regions will be included in the backup.
{% endhint %}

3. Verify backup contents:

{% tabs %}
{% tab title="Bash" %}

```bash
tar tvf /tmp/backup-mgmt/backup.tar.gz
```

{% endtab %}
{% endtabs %}

The backup archive should contain:

* `state_backup.yaml`: System state configuration
* `kplane_values_backup.yaml`: Kubernetes management cluster configuration
* `consul.snap`: Consul snapshot
* `mysql_dump_Infra.sql`: Infrastructure database backup
* `mysql_dump_Region1.sql`: Region-specific database backup
* `ovn-north-backup` & `ovn-south-backup` : Ovn database backup

{% hint style="warning" %}
**Warning**

Metrics service (gnocci) persistent data will not be backed up or restored as part of the above procedure. Therefore, in a full disaster recovery scenario, you must manually copy the metrics service (gnocchi) metrics data from the original storage class `pcd-sc` persistent volume.
{% endhint %}

## Automated Backup Configuration

The <code class="expression">space.vars.product\_name</code> management plane includes an automated backup system that protects your data and configuration. This system creates regular backups and can store them both locally and in Amazon S3.

Learn how to verify backup operations and configure S3 storage for your backups.

### Understanding the backup system

When you install <code class="expression">space.vars.product\_name</code> management plane, the system automatically sets up backup protection for you. During installation, it creates a cronjob called `mgmt-plane-backup` in the `pf9-utils` namespace that runs every hour to back up your system.

Your backups get stored in a dedicated storage area called `mgmt-plane-backup-pvc` on your Kubernetes cluster. This storage persists even if pods restart, keeping your backup data safe and accessible.

#### Step 1: Verify backup operations

You can check the status of your backup system at any time using kubectl commands.

1. Run the following command to view the backup cronjob status.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get cronjob mgmt-plane-backup -n pf9-utils
```

{% endtab %}
{% endtabs %}

This displays when you ran the last backup, confirming your system is working properly.

#### Step 2: Check backup logs

To troubleshoot backup operations or verify successful completion, you can view the backup logs.

1. List the backup pods to find the most recent operation.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get po -n pf9-utils | grep mgmt-plane-backup
```

{% endtab %}
{% endtabs %}

2. From the output, copy the pod name with the most recent timestamp.
3. View the logs for that specific pod.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl logs mgmt-plane-backup-29167800-l294f -n pf9-utils
```

{% endtab %}
{% endtabs %}

Replace `mgmt-plane-backup-29167800-l294f`with your actual pod name from step 1.

The logs show detailed information about the backup operation, including any errors or success messages.

#### Step 3: Configure S3 backup storage

To enable storing backups in an S3 bucket, you need to configure S3 credentials in a secret named `aws-credentials` in the`pf9-utils` namespace.

Before you begin, consider the following points.

* The Kubernetes secret should have appropriate access permissions for the `mgmt-plane-backup` cronjob in the `pf9-utils` namespace.
* The backup system stores data in the PVC named `mgmt-plane-backup-pvc` on the Kubernetes cluster and will also upload to your configured S3 location.
* The `mgmt-plane-backup`cronjob runs hourly to ensure regular system backups to both local storage and S3.

1. Run the following command to open and edit `aws-credentials` on the editor.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl edit secret aws-credentials -n pf9-utils
```

{% endtab %}
{% endtabs %}

2. On the `aws-credentials` edit section `data:`

{% tabs %}
{% tab title="Bash" %}

```bash
#Make the edits the yaml file

data:
AWS_ACCESS_KEY_ID: "<YOUR_ACCESS_KEY>"

AWS_SECRET_ACCESS_KEY: "<YOUR_SECRET_KEY>"

AWS_REGION: "<YOUR_AWS_REGION>"

AWSS3 PATH: "s3://<YOUR_BUCKET_NAME/PATH/>"
```

{% endtab %}
{% endtabs %}

3. Replace the placeholder values with your actual AWS credentials.

| Placeholder              | Replace with                             |
| ------------------------ | ---------------------------------------- |
| `YOUR_ACCESS_KEY`        | Your AWS access key ID                   |
| `YOUR_SECRET_KEY`        | Your AWS secret access key               |
| `YOUR_AWS_REGION`        | Your AWS region (for example, us-west-2) |
| `YOUR_BUCKET_NAME/PATH/` | Your S3 bucket name and optional path    |

3. Save and close the editor.

Once configured, your backups will be stored both locally in the PVC and in your specified S3 bucket location. This provides enhanced data protection and allows for disaster recovery scenarios.

You have successfully configured automated backups for your <code class="expression">space.vars.product\_name</code> management plane. Your system now creates regular backups and stores them securely both locally and in Amazon S3.

## Manual Restore Procedure

When you need to restore your <code class="expression">space.vars.product\_name</code> management plane, you can access backups stored locally in the PVC or from Amazon S3. This section walks you through both restoration methods.

### Standard Restore (from local PVC)

To restore from local backups, you need to access the backup files stored in the mgmt-plane-backup-pvc. This process involves finding the PVC, locating the underlying storage, and mounting it to access the backup files.

#### Step 1: Locate the backup PVC

Run the following command to find the backup PVC in the `pf9-utils` namespace.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get pvc -n pf9-utils
```

{% endtab %}
{% endtabs %}

The output displays your backup PVC details. Here is a sample example.

{% tabs %}
{% tab title="Example" %}

```bash
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
mgmt-plane-backup-pvc   Bound    pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632   100Gi      RWO            pcd-sc         <unset>                 34h
```

{% endtab %}
{% endtabs %}

#### Step 2: Find the underlying persistent volume

Get the volume name for the backup PVC, by running the following command.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get pvc mgmt-plane-backup-pvc -n pf9-utils -o jsonpath='{.spec.volumeName}'
```

{% endtab %}
{% endtabs %}

The output returns the persistent volume name. Here is an example.

{% tabs %}
{% tab title="Bash" %}

```bash
pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
```

{% endtab %}
{% endtabs %}

#### Step 3: Get the persistent volume configuration

Describe the persistent volume to find the NFS share information by running the following command.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl describe pv pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
```

{% endtab %}
{% endtabs %}

In the output, look for the `Source:` section with `VolumeHandle` field. This contains the NFS server and path information that you need for mounting.

#### Step 4: Install NFS utilities

Install the required NFS packages on your system by running the following command.

{% tabs %}
{% tab title="Bash" %}

```bash
ubuntu@sample-vm:~$ sudo apt update && sudo apt -y install nfs-common
```

{% endtab %}
{% endtabs %}

#### Step 5: Mount the NFS share

Create a local directory and mount the NFS share:

{% tabs %}
{% tab title="Bash" %}

```bash
ubuntu@sample-vm:~$ mkdir -p /home/ubuntu/backups
ubuntu@sample-vm:~$ sudo mount -t nfs 10.149.106.253:/mnt/gnocchi /home/ubuntu/backups
```

{% endtab %}
{% endtabs %}

Replace `10.149.106.253:/mnt/gnocchi` with the server and share path from your `VolumeHandle` field.

#### Step 6: Access the backup files

List the available backup files by running the following command

{% tabs %}
{% tab title="Bash" %}

```bash
ubuntu@sample-vm:~$ ls backups/pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
```

{% endtab %}
{% endtabs %}

The backup files will display with timestamps. Here is an sample output.

{% tabs %}
{% tab title="Bash" %}

```bash
backup_20250608_210055.tar.gz  backup_20250608_230057.tar.gz  backup_20250609_010057.tar.gz
backup_20250609_030054.tar.gz  backup_20250609_050055.tar.gz  backup_20250608_220057.tar.gz
```

{% endtab %}
{% endtabs %}

Choose the backup file you want to restore and proceed with your restoration process. Ensure you are using the appropriate restoration commands for your specific backup file.

### Restore from S3 Backup

Create and configure the `/etc/default/airctl-backup` file with required AWS parameters, making sure that `AWS_S3_PATH` points specifically to the backup file you want to restore, not just the S3 bucket:

{% tabs %}
{% tab title="Bash" %}

```bash
AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY>
AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_KEY>
AWS_REGION=<YOUR_AWS_REGION>
AWS_S3_PATH=s3://<YOUR_BUCKET_NAME/PATH/SPECIFIC_BACKUP_FILE>
```

{% endtab %}
{% endtabs %}

Execute the S3 restore command:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl restore --s3backup --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**NOTE**

For complete disaster recovery, manually restore Gnocchi metrics data from the original `pcd-sc` persistent volume
{% endhint %}

Verification Steps

Check backup file integrity using MD5 checksum::

{% tabs %}
{% tab title="Bash" %}

```bash
# Generate MD5 checksum for the backup file
md5sum <BACKUP_FILE>.tar.gz

# Optional: Compare with a pre-recorded checksum
# You can save the MD5 checksum when initially creating the backup
md5sum /root/backup-mgmt/backup.tar.gz > backup-checksum.txt

# Later, verify the backup file matches the original checksum
md5sum -c backup-checksum.txt
```

{% endtab %}
{% endtabs %}

Verify S3 uploads (if configured):

{% tabs %}
{% tab title="Bash" %}

```bash
aws s3 ls s3://<BUCKET_NAME>/<BACKUP_PATH>
```

{% endtab %}
{% endtabs %}

Monitor restore progress:

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl logs -f <RESTORE_POD_NAME>
```

{% endtab %}
{% endtabs %}

## Common Issues

* If AWS credentials are not properly configured, automated S3 backups will continue locally but skip S3 upload
* Restore operations may take significant time depending on data volume
* Services may take additional time to start after restore completion