--- title: Automated disaster recovery for a hosted cluster by using OADP --- # Automated disaster recovery for a hosted cluster by using OADP {#hcp-disaster-recovery-oadp-auto} In hosted clusters on bare-metal or Amazon Web Services (AWS) platforms, you can automate some backup and restore steps by using the OpenShift API for Data Protection (OADP) Operator. The process involves the following steps: 1. Configuring OADP 2. Defining a Data Protection Application (DPA) 3. Backing up the data plane workload 4. Backing up the control plane workload 5. Restoring a hosted cluster by using OADP ## Prerequisites to automate disaster recovery by using OADP {#hcp-dr-oadp-auto-prereqs_hcp-disaster-recovery-oadp-auto} Ensure that you meet the prerequisites to automate disaster recovery for hosted control planes by using OADP. The following prerequisites apply to the management cluster: - You installed the OADP Operator. For more information, see "About installing OADP". - You created a storage class. - You have access to the cluster with `cluster-admin` privileges. - You have access to the OADP subscription through a catalog source. - You have access to a cloud storage provider that is compatible with OADP, such as S3, Microsoft Azure, Google Cloud, or MinIO. - In a disconnected environment, you have access to a self-hosted storage provider that is compatible with OADP, for example Red Hat OpenShift Data Foundation or MinIO. - Your hosted control planes pods are up and running. - You are using a supported version of OADP for your management cluster. For example, if your management cluster is on OpenShift Container Platform 4.20, you must use OADP version 1.5. For more information, see "Support for OpenShift API for Data Protection (OADP)". **Additional resources** {._additional-resources} - [About installing OADP](/openshift-docs-markdown/backup_and_restore/application_backup_and_restore/installing/about-installing-oadp#about-installing-oadp) - [Red Hat OpenShift Data Foundation](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/) - [MinIO](https://min.io/) - [Support for OpenShift API for Data Protection (OADP)](/openshift-docs-markdown/backup_and_restore/application_backup_and_restore/oadp-intro#oadp-operator-supported_oadp-api) ## Configuring OADP to automate disaster recovery for hosted control planes {#hcp-dr-prep-oadp-auto_hcp-disaster-recovery-oadp-auto} Before you can automate disaster recovery by using OpenShift API for Data Protection (OADP), you need to configure it for your hosted control planes platform. **Procedure** - If your hosted cluster is on AWS, follow the steps in "Configuring the OpenShift API for Data Protection with AWS S3 compatible storage" to configure OADP. - If your hosted cluster is on a bare metal, follow the steps in "Configuring the OpenShift API for Data Protection with Multicloud Object Gateway" to configure OADP. **Additional resources** {._additional-resources} - [Configuring the OpenShift API for Data Protection with AWS S3 compatible storage](/openshift-docs-markdown/backup_and_restore/application_backup_and_restore/installing/installing-oadp-mcg#installing-oadp-mcg) - [Configuring the OpenShift API for Data Protection with Multicloud Object Gateway](/openshift-docs-markdown/backup_and_restore/application_backup_and_restore/installing/installing-oadp-aws#installing-oadp-aws) ## Automation of the backup and restore process with a DPA {#hcp-dr-oadp-dpa_hcp-disaster-recovery-oadp-auto} You can automate parts of the backup and restore process by using a Data Protection Application (DPA). When you use a DPA, the steps to pause and restart the reconciliation of resources are automated. The DPA defines information including backup locations and Velero pod configurations. ### Creating a Data Protection Application for bare metal {#hcp-dr-oadp-dpa-bm_hcp-disaster-recovery-oadp-auto} Automate parts of the backup and restore process on bare metal by creating a Data Protection Application (DPA). A DPA defines information including backup locations and Velero pod configurations. You can create a DPA by defining a `DataProtectionApplication` object. **Procedure** 1. Create a manifest file similar to the following example: ```yaml apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample namespace: openshift-adp spec: backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: prefix: config: region: minio profile: "default" s3ForcePathStyle: "true" s3Url: "" insecureSkipTLSVerify: "true" credential: key: cloud name: cloud-credentials default: true snapshotLocations: - velero: provider: aws config: region: minio profile: "default" credential: key: cloud name: cloud-credentials configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - openshift - aws - csi - hypershift resourceTimeout: 2h ``` - `spec.backupLocations.velero.provider` specifies the provider for Velero. If you are using bare metal and MinIO, you can use `aws` as the provider. - `spec.backupLocations.velero.objectStorage.bucket` specifies the bucket name; for example, `oadp-backup`. - `spec.backupLocations.velero.objectStorage.prefix` specifies the bucket prefix; for example, `hcp`. - `spec.backupLocations.velero.config.region` specifies the bucket region. In this example, the region is `minio`, which is a storage provider that is compatible with the S3 API. - `spec.backupLocations.velero.config.s3Url` specifies the URL of the S3 endpoint. - `spec.snapshotLocations.velero.provider` specifies the provider for Velero. If you are using bare metal and MinIO, you can use `aws` as the provider. - `spec.snapshotLocations.velero.config.region` specifies the region. In this example, the region is `minio`, which is a storage provider that is compatible with the S3 API. - `spec.configuration.nodeAgent.uploaderType` specifies `kopia` as the uploader type. The `restic` uploader type is deprecated for OADP 1.5 and later. 2. Create the DPA object by running the following command: ```terminal $ oc create -f dpa.yaml ``` After you create the `DataProtectionApplication` object, new `velero` deployment and `node-agent` pods are created in the `openshift-adp` namespace. **Next steps** - Back up the data plane workload. ### Creating a Data Protection Application for AWS {#hcp-dr-oadp-dpa-aws_hcp-disaster-recovery-oadp-auto} Automate parts of the backup and restore process on AWS by creating a Data Protection Application (DPA). A DPA defines information including backup locations and Velero pod configurations. You can create a DPA by defining a `DataProtectionApplication` object. **Procedure** 1. Create a manifest file similar to the following example: ```yaml apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample namespace: openshift-adp spec: backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: prefix: config: region: minio profile: "backupStorage" credential: key: cloud name: cloud-credentials snapshotLocations: - velero: provider: aws config: region: minio profile: "volumeSnapshot" credential: key: cloud name: cloud-credentials configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - openshift - aws - csi - hypershift resourceTimeout: 2h ``` - `spec.backupLocations.velero.objectStorage.bucket` specifies the bucket name; for example, `oadp-backup`. - `spec.backupLocations.velero.objectStorage.prefix` specifies the bucket prefix; for example, `hcp`. - `spec.backupLocations.velero.config.region` specifies the bucket region. The bucket region in this example is `minio`, which is a storage provider that is compatible with the S3 API. - `spec.snapshotLocations.velero.config.region` specifies the region. The region in this example is `minio`, which is a storage provider that is compatible with the S3 API. - `spec.configuration.nodeAgent.uploaderType` specifies `kopia` as the uploader type. The `restic` uploader type is deprecated for OADP 1.5 and later. 2. Create the DPA resource by running the following command: ```terminal $ oc create -f dpa.yaml ``` After you create the `DataProtectionApplication` object, new `velero` deployment and `node-agent` pods are created in the `openshift-adp` namespace. **Next steps** - Back up the data plane workload. ## Backing up the data plane workload by using the OADP Operator {#hcp-dp-backup-oadp-auto_hcp-disaster-recovery-oadp-auto} You can back up the data plane workload by using the OADP Operator. However, if the data plane workload is not important, you can skip this procedure. **Procedure** - To back up the data plane workload, follow the steps in "Backing up applications". **Additional resources** {._additional-resources} - [Backing up applications](/openshift-docs-markdown/backup_and_restore/application_backup_and_restore/backing_up_and_restoring/backing-up-applications#backing-up-applications) ## Backing up the control plane workload {#hcp-dr-oadp-backup-cp-workload-auto_hcp-disaster-recovery-oadp-auto} You can back up the control plane workload by creating the `Backup` custom resource (CR). To monitor and observe the backup process, see "Observing the backup and restore process". **Procedure** 1. Create a YAML file that defines the `Backup` CR: ```yaml apiVersion: velero.io/v1 kind: Backup metadata: name: namespace: openshift-adp labels: velero.io/storage-location: default spec: hooks: {} includedNamespaces: - - includedResources: - sa - role - rolebinding - pod - pvc - pv - bmh - configmap - infraenv - priorityclasses - pdb - agents - hostedcluster - nodepool - secrets - services - deployments - hostedcontrolplane - cluster - agentcluster - agentmachinetemplate - agentmachine - machinedeployment - machineset - machine - route - clusterdeployment excludedResources: [] storageLocation: default ttl: 2h0m0s snapshotMoveData: true datamover: "velero" defaultVolumesToFsBackup: false ``` - `metadata.name` specifies the name for your `Backup` resource. - `spec.includedNamespaces` specifies namespaces to back up objects from. You must replace `` with the name of the hosted cluster namespace and replace `` with the name of the hosted control plane namespace. - `spec.includedResources` includes the `infraenv` value. You must create the `infraenv` resource in a separate namespace. Do not delete the `infraenv` resource during the backup process. - `spec.snapshotMoveData: true` and `spec.datamover: velero` enable the CSI volume snapshots and upload the control plane workload automatically to cloud storage. - `spec.defaultVolumesToFsBackup` specifies that the `fs-backup` backing up method for persistent volumes (PVs) is not used. > [!NOTE] > If you want to use CSI volume snapshots, you must add the `backup.velero.io/backup-volumes-excludes=` annotation to your PVs. 2. Apply the `Backup` CR by running the following command: ```terminal $ oc apply -f backup-control-plane.yaml ``` **Verification** - Verify that the value of the `status.phase` is `Completed` by running the following command: ```terminal $ oc get backups.velero.io -n openshift-adp \ -o jsonpath='{.status.phase}' ``` **Next steps** - Restore the hosted cluster by using OADP. ## Restoring a hosted cluster by using OADP {#hcp-dr-oadp-restore-auto_hcp-disaster-recovery-oadp-auto} You can restore the hosted cluster by creating the `Restore` custom resource (CR). - If you are using an in-place update, the `InfraEnv` resource does not need spare nodes. You need to re-provision the worker nodes from the new management cluster. - If you are using a replace update, you need some spare nodes for the `InfraEnv` resource to deploy the worker nodes. > [!IMPORTANT] > After you back up your hosted cluster, you must delete it to start the restoring process. To start node provisioning, you must back up workloads in the data plane before deleting the hosted cluster. **Prerequisites** - You completed the steps in [Removing a cluster by using the console](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.13/html/clusters/cluster_mce_overview#remove-a-cluster-by-using-the-console) (RHACM documentation) to delete your hosted cluster. - You completed the steps in [Removing remaining resources after removing a cluster](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.13/html/clusters/cluster_mce_overview#removing-a-cluster-from-management-in-special-cases) (RHACM documentation). To monitor and observe the backup process, see "Observing the backup and restore process". **Procedure** 1. Verify that no pods and persistent volume claims (PVCs) are present in the hosted control plane namespace by running the following command: ```terminal $ oc get pod pvc -n ``` ```terminal {title="Expected output"} No resources found ``` 2. Create a YAML file that defines the `Restore` CR: ```yaml apiVersion: velero.io/v1 kind: Restore metadata: name: namespace: openshift-adp spec: backupName: restorePVs: true existingResourcePolicy: update excludedResources: - nodes - events - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io ``` - `metadata.name` specifies the name for your `Restore` resource. - `spec.backupName` specifies the name of your `Backup` resource. - `spec.restorePVs: true` indicates the recovery of persistent volumes (PVs) and their pods. - `spec.existingResourcePolicy: update` ensures that the existing objects are overwritten with the backed up content. > [!IMPORTANT] > You must create the `InfraEnv` resource in a separate namespace. Do not delete the `InfraEnv` resource during the restore process. The `InfraEnv` resource is mandatory for the new nodes to be reprovisioned. 3. Apply the `Restore` CR by running the following command: ```terminal $ oc apply -f restore-hosted-cluster.yaml ``` 4. Verify if the value of the `status.phase` is `Completed` by running the following command: ```terminal $ oc get hostedcluster -n \ -o jsonpath='{.status.phase}' ``` ## Observing the backup and restore process {#hcp-dr-oadp-observe_hcp-disaster-recovery-oadp-auto} When you use OpenShift API for Data Protection (OADP) to back up and restore a hosted cluster, you can monitor and observe the process. **Procedure** 1. Observe the backup process by running the following command: ```terminal $ watch "oc get backups.velero.io -n openshift-adp -o jsonpath='{.status}'" ``` 2. Observe the restore process by running the following command: ```terminal $ watch "oc get restores.velero.io -n openshift-adp -o jsonpath='{.status}'" ``` 3. Observe the Velero logs by running the following command: ```terminal $ oc logs -n openshift-adp -ldeploy=velero -f ``` 4. Observe the progress of all of the OADP objects by running the following command: ```terminal $ watch "echo BackupRepositories:;echo;oc get backuprepositories.velero.io -A;echo; echo BackupStorageLocations: ;echo; oc get backupstoragelocations.velero.io -A;echo;echo DataUploads: ;echo;oc get datauploads.velero.io -A;echo;echo DataDownloads: ;echo;oc get datadownloads.velero.io -n openshift-adp; echo;echo VolumeSnapshotLocations: ;echo;oc get volumesnapshotlocations.velero.io -A;echo;echo Backups:;echo;oc get backup -A; echo;echo Restores:;echo;oc get restore -A" ``` ## Using the OADP CLI to describe the Backup and Restore resources {#hcp-dr-oadp-observe-velero_hcp-disaster-recovery-oadp-auto} When you use OpenShift API for Data Protection, you can get more details of the `Backup` and `Restore` resources by using the OADP command-line interface (CLI). **Procedure** 1. Get details of your `Restore` custom resource (CR) by running the following command: ```terminal $ oc oadp restore describe --details ``` Replace `` with the name of your `Restore` resource. 2. Get details of your `Backup` CR by running the following command: ```terminal $ oc oadp backup describe --details ``` Replace `` with the name of your `Backup` resource.