# Backup and Restore Management Plane

This guide provides steps for backing up and restoring the self-hosted <code class="expression">space.vars.product\_name</code> management plane in disaster recovery scenarios. The procedures include both manual and automated backup methods, as well as manual restoration process.

{% hint style="info" %}
**Info**

When restoring the management plane, ensure it's done on a Kubernetes cluster that is separate from the cluster where the backup was generated.
{% endhint %}

## Prerequisites

#### System Requirements

* Access to the Kubernetes management cluster
* Installed and configured `airctl` binary
* Valid `airctl` configuration file at `/opt/pf9/airctl/conf/airctl-config.yaml`
* Root or sudo access to the management node

#### For S3 Backup Storage

* AWS credentials with S3 bucket access
* Existing S3 bucket for backup storage
* AWS CLI configured (for verification purposes)

## Important Considerations

1. The restoration process must be performed on a separate Kubernetes management cluster that is different from the management cluster where the backup was generated.
2. The metrics service (gnocci) data is not backed up in the backup procedure. For complete disaster recovery, manual copying of metrics service (gnocchi) metrics from the original storage class `pcd-sc` persistent volume is required.

## Manual Backup Procedure

Create a backup directory:

{% tabs %}
{% tab title="Bash" %}

```bash
mkdir -p /root/backup-mgmt/
```

{% endtab %}
{% endtabs %}

Execute the airctl backup command:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl backup --outdir /root/backup-mgmt/ --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```

{% endtab %}
{% endtabs %}

Verify backup contents:

{% tabs %}
{% tab title="Bash" %}

```bash
tar tvf /root/backup-mgmt/backup.tar.gz
```

{% endtab %}
{% endtabs %}

The backup archive should contain:

* `state_backup.yaml`: System state configuration
* `kplane_values_backup.yaml`: Kubernetes management cluster configuration
* `consul.snap`: Consul snapshot
* `mysql_dump_Infra.sql`: Infrastructure database backup
* `mysql_dump_Region1.sql`: Region-specific database backup
* `ovn-north-backup` & `ovn-south-backup` : Ovn database backup

{% hint style="warning" %}
**Warning**

Metrics service (gnocci) persistent data will not be backed up or restored as part of the above procedure. Therefore, in a full disaster recovery scenario, you must manually copy the metrics service (gnocchi) metrics data from the original storage class `pcd-sc` persistent volume.
{% endhint %}

## Automated Backup Configuration

The automated backup system is created during the initial [installation](https://docs.platform9.com/private-cloud-director/2025.2/getting-started/self-hosted-install#install-management-plane) of the <code class="expression">space.vars.product\_name</code> management plane. When you run the installation command, the system automatically creates a `CronJob` named `mgmt-plane-backup`. This cronjob is designed to run `every hour` to ensure regular system backups.

Backups are stored at path /var/pf9/backups/ on the node with `airctl` installation

{% tabs %}
{% tab title="Bash" %}

```bash
$ kubectl get cronjob 
NAME                SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
mgmt-plane-backup   0 * * * *   False     0        2m56s           12m
```

{% endtab %}
{% endtabs %}

#### Configuring S3 Backup Storage

Locate the `aws-credentials` secret, Empty secret `aws-credentials` created by default::

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get secret aws-credentials -o yaml | grep AWS
```

{% endtab %}
{% endtabs %}

Configure the secret with required AWS parameters:

* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`
* `AWS_S3_PATH`

{% hint style="info" %}
**Info**

If the secret is patched with the required AWS credentials and S3 bucket information, the cronjob execution will store backups locally and push them to the S3 bucket.
{% endhint %}

Once configured, backups will be stored both locally and in the specified S3 bucket.

## Manual Restore Procedure

#### Standard Restore

Execute the restore command:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl restore --backupdir /root/backup-mgmt --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```

{% endtab %}
{% endtabs %}

#### Restore from S3 Backup

1. Create and configure the `aws-credentials` secret with required AWS parameters
2. Execute the S3 restore command

{% tabs %}
{% tab title="Bash" %}

```bash
airctl restore --s3backup --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

For complete disaster recovery, manually restore Gnocchi metrics data from the original `pcd-sc` persistent volume
{% endhint %}

## Verification Steps

Check backup file integrity using MD5 checksum::

{% tabs %}
{% tab title="Bash" %}

```bash
# Generate MD5 checksum for the backup file
md5sum <backup-file>.tar.gz

# Optional: Compare with a pre-recorded checksum
# You can save the MD5 checksum when initially creating the backup
md5sum /root/backup-mgmt/backup.tar.gz > backup-checksum.txt

# Later, verify the backup file matches the original checksum
md5sum -c backup-checksum.txt
```

{% endtab %}
{% endtabs %}

Verify S3 uploads (if configured):

{% tabs %}
{% tab title="Bash" %}

```bash
aws s3 ls s3://<bucket-name>/<backup-path>
```

{% endtab %}
{% endtabs %}

Monitor restore progress:

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl logs -f <restore-pod-name>
```

{% endtab %}
{% endtabs %}

## Common Issues

* If AWS credentials are not properly configured, automated S3 backups will continue locally but skip S3 upload
* Restore operations may take significant time depending on data volume
* Services may take additional time to start after restore completion