--- title: Backing up and restoring etcd data --- # Backing up and restoring etcd data {#backup-etcd} You can back up etcd data for your OpenShift Container Platform cluster by manually creating an etcd snapshot and saving static pod resources on a control plane host, or by configuring automated single or recurring backups. etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects. Back up etcd data for your cluster regularly. Store backups in a secure location, ideally outside the OpenShift Container Platform environment. Do not take an etcd backup before the first certificate rotation completes. Certificate rotation occurs 24 hours after installation. A backup taken before rotation completes contains expired certificates. Take etcd backups during non-peak usage hours when possible. The etcd snapshot has a high I/O cost. Be sure to take an etcd backup before you update your cluster. Taking a backup before you update is important because when you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OpenShift Container Platform 4.17.5 cluster must use an etcd backup that was taken from 4.17.5. > [!IMPORTANT] > Back up etcd data for your cluster by performing a single invocation of the backup script on a control plane host. Do not take a backup for each control plane host. ## Backing up etcd data {#backing-up-etcd-data_backup-etcd} You can back up etcd data by creating an etcd snapshot and saving the static pod resources on a control plane host. This backup preserves the cluster state and provides the resources required to restore etcd at a later time. > [!IMPORTANT] > Only save a backup from a single control plane host. Do not take a backup from each control plane host in the cluster. For a Two-Node with Fencing (TNF) setup, follow the steps to back up etcd data on only one node in the cluster. The cluster restore process is driven by data from a single node, so you can perform the etcd backup steps on only one node. **Prerequisites** - You have access to the cluster as a user with the `cluster-admin` role. - You have verified whether the cluster-wide proxy is enabled. > [!TIP] > You can check whether the proxy is enabled by reviewing the output of `oc get proxy cluster -o yaml`. The proxy is enabled if the `httpProxy`, `httpsProxy`, and `noProxy` fields have values set. **Procedure** 1. Start a debug session as root for a control plane node: ```terminal $ oc debug --as-root node/ ``` 2. Change your root directory to `/host` in the debug shell: ```terminal sh-4.4# chroot /host ``` 3. If the cluster-wide proxy is enabled, export the `NO_PROXY`, `HTTP_PROXY`, and `HTTPS_PROXY` environment variables by running the following commands: ```terminal $ export HTTP_PROXY=http://:8080 ``` ```terminal $ export HTTPS_PROXY=https://:8080 ``` ```terminal $ export NO_PROXY= ``` 4. Run the `cluster-backup.sh` script with the path to the directory where you want to save the backup: > [!TIP] > The `cluster-backup.sh` script is maintained as a component of the etcd Cluster Operator and is a wrapper around the `etcdctl snapshot save` command. ```terminal sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup ``` ```terminal {title="Example script output"} found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6 found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-7 found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-6 found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3 ede95fe6b88b87ba86a03c15e669fb4aa5bf0991c180d3c6895ce72eaade54a1 etcdctl version: 3.4.14 API version: 3.4 {"level":"info","ts":1624647639.0188997,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db.part"} {"level":"info","ts":"2021-06-25T19:00:39.030Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"} {"level":"info","ts":1624647639.0301006,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.0.5:2379"} {"level":"info","ts":"2021-06-25T19:00:40.215Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"} {"level":"info","ts":1624647640.6032252,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.0.5:2379","size":"114 MB","took":1.584090459} {"level":"info","ts":1624647640.6047094,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db"} Snapshot saved at /home/core/assets/backup/snapshot_2021-06-25_190035.db {"hash":3866667823,"revision":31407,"totalKey":12828,"totalSize":114446336} snapshot db and kube resources are successfully saved to /home/core/assets/backup ``` In this example, two files are created in the `/home/core/assets/backup/` directory on the control plane host: - `snapshot_.db`: This file is the etcd snapshot. The `cluster-backup.sh` script confirms the validity of the snapshot. - `static_kuberesources_.tar.gz`: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot. > [!NOTE] > If etcd encryption is enabled, store this second file separately from the etcd snapshot for security reasons. However, this file is required to restore from the etcd snapshot. > > The etcd encryption only encrypts values, not keys. This means that resource types, namespaces, and object names are not encrypted. **Additional resources** {._additional-resources} - [Restoring to an earlier cluster state](/openshift-docs-markdown/backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state#dr-restoring-cluster-state) - [Recovering an unhealthy etcd cluster for hosted control planes](/openshift-docs-markdown/hosted_control_planes/hcp_high_availability/hcp-recovering-etcd-cluster#hcp-recovering-etcd-cluster) ## Creating automated etcd backups {#creating-automated-etcd-backups_backup-etcd} You can enable automated etcd backups for your cluster by applying a `FeatureGate` and backup custom resources (CRs). > [!IMPORTANT] > Automating etcd backups is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. > > For more information about the support scope of Red Hat Technology Preview features, see [Technology Preview Features Support Scope](https://access.redhat.com/support/offerings/techpreview/). > [!WARNING] > Enabling the `TechPreviewNoUpgrade` feature set on your cluster prevents minor version updates. The `TechPreviewNoUpgrade` feature set cannot be disabled. Do not enable this feature set on production clusters. **Prerequisites** - You have access to the cluster as a user with the `cluster-admin` role. - You have access to the OpenShift CLI (`oc`). **Procedure** 1. Create a `FeatureGate` custom resource (CR) file named `enable-tech-preview-no-upgrade.yaml` with the following contents: ```yaml apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: name: cluster spec: featureSet: TechPreviewNoUpgrade ``` 2. Apply the CR by running the following command: ```terminal $ oc apply -f enable-tech-preview-no-upgrade.yaml ``` Applying the `FeatureGate` enables the automated backup APIs. It takes time for the related APIs to become available. 3. Verify that the custom resource definition (CRD) was created by running the following command: ```terminal $ oc get crd | grep backup ``` ```terminal {title="Example output"} backups.config.openshift.io 2023-10-25T13:32:43Z etcdbackups.operator.openshift.io 2023-10-25T13:32:04Z ``` ### Creating a single automated etcd backup {#creating-single-etcd-backup_backup-etcd} You can create a single automated etcd backup by applying an `EtcdBackup` custom resource (CR). Backup data is stored on either dynamically-provisioned or local storage. **Prerequisites** - You have access to the cluster as a user with the `cluster-admin` role. - You have access to the OpenShift CLI (`oc`). **Procedure** 1. If dynamically-provisioned storage is available, complete the following steps to create a single automated etcd backup: 1. Create a persistent volume claim (PVC) named `etcd-backup-pvc.yaml` with contents such as the following example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: etcd-backup-pvc namespace: openshift-etcd spec: accessModes: - ReadWriteOnce resources: requests: storage: volumeMode: Filesystem ``` where: `` : Specifies the amount of storage available to the PVC. Adjust this value for your requirements, such as `200Gi`. 2. Apply the PVC by running the following command: ```terminal $ oc apply -f etcd-backup-pvc.yaml ``` 3. Verify that the PVC was created by running the following command: ```terminal $ oc get pvc ``` ```terminal {title="Example output"} NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE etcd-backup-pvc Bound 51s ``` > [!NOTE] > Dynamic PVCs stay in the `Pending` state until they are mounted. 4. Create a CR file named `etcd-single-backup.yaml` with contents such as the following example: ```yaml apiVersion: operator.openshift.io/v1alpha1 kind: EtcdBackup metadata: name: etcd-single-backup namespace: openshift-etcd spec: pvcName: ``` where: `` : Specifies the name of the PVC to save the backup to. Adjust this value according to your environment, such as `etcd-backup-pvc`. 5. Apply the CR to start a single backup by running the following command: ```terminal $ oc apply -f etcd-single-backup.yaml ``` 2. If dynamically-provisioned storage is not available, complete the following steps to create a single automated etcd backup: 1. Create a `StorageClass` CR file named `etcd-backup-local-storage.yaml` with the following contents: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: etcd-backup-local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: Immediate ``` 2. Apply the `StorageClass` CR by running the following command: ```terminal $ oc apply -f etcd-backup-local-storage.yaml ``` 3. Create a PV named `etcd-backup-pv-fs.yaml` with contents such as the following example: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: etcd-backup-pv-fs spec: capacity: storage: volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: etcd-backup-local-storage local: path: /mnt nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ``` where: `` : Specifies the amount of storage available to the PV. Adjust this value for your requirements, such as `100Gi`. `` : Specifies the control plane node to attach this PV to. Replace with the actual node name. 4. Verify that the PV was created by running the following command: ```terminal $ oc get pv ``` ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE etcd-backup-pv-fs 100Gi RWO Retain Available etcd-backup-local-storage 10s ``` 5. Create a PVC named `etcd-backup-pvc.yaml` with contents such as the following example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: etcd-backup-pvc namespace: openshift-etcd spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: ``` where: `` : Specifies the amount of storage available to the PVC. Adjust this value for your requirements, such as `10Gi`. 6. Apply the PVC by running the following command: ```terminal $ oc apply -f etcd-backup-pvc.yaml ``` 7. Create a CR file named `etcd-single-backup.yaml` with contents such as the following example: ```yaml apiVersion: operator.openshift.io/v1alpha1 kind: EtcdBackup metadata: name: etcd-single-backup namespace: openshift-etcd spec: pvcName: ``` where: `` : Specifies the name of the PVC to save the backup to. Adjust this value according to your environment, such as `etcd-backup-pvc`. 8. Apply the CR to start a single backup by running the following command: ```terminal $ oc apply -f etcd-single-backup.yaml ``` ### Creating recurring automated etcd backups {#creating-recurring-etcd-backups_backup-etcd} You can create recurring automated etcd backups by applying a custom resource (CR) that defines a backup schedule and retention policy. Backup data is stored on either dynamically-provisioned or local storage. Use dynamically-provisioned storage to keep the created etcd backup data in a safe, external location if possible. If dynamically-provisioned storage is not available, consider storing the backup data on an NFS share to make backup recovery more accessible. **Prerequisites** - You have access to the cluster as a user with the `cluster-admin` role. - You have access to the OpenShift CLI (`oc`). **Procedure** 1. If dynamically-provisioned storage is available, complete the following steps to create automated recurring backups: 1. Create a persistent volume claim (PVC) named `etcd-backup-pvc.yaml` with contents such as the following example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: etcd-backup-pvc namespace: openshift-etcd spec: accessModes: - ReadWriteOnce resources: requests: storage: 200Gi volumeMode: Filesystem storageClassName: etcd-backup-local-storage ``` where: `spec.resources.requests.storage` : Specifies the amount of storage available to the PVC. Adjust this value for your requirements. > [!NOTE] > Each of the following providers requires changes to the `accessModes` and `storageClassName` keys: > > | Provider | `accessModes` value | `storageClassName` value | > | --- | --- | --- | > | AWS with the `versioned-installer-efc_operator-ci` profile | `- ReadWriteMany` | `efs-sc` | > | Google Cloud | `- ReadWriteMany` | `filestore-csi` | > | Microsoft Azure | `- ReadWriteMany` | `azurefile-csi` | 2. Apply the PVC by running the following command: ```terminal $ oc apply -f etcd-backup-pvc.yaml ``` 3. Verify that the PVC was created by running the following command: ```terminal $ oc get pvc ``` ```terminal {title="Example output"} NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE etcd-backup-pvc Bound 51s ``` > [!NOTE] > Dynamic PVCs stay in the `Pending` state until they are mounted. 2. If dynamically-provisioned storage is unavailable, create a local storage PVC by completing the following steps: > [!WARNING] > If you delete or otherwise lose access to the node that contains the stored backup data, you can lose data. 1. Create a `StorageClass` CR file named `etcd-backup-local-storage.yaml` with the following contents: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: etcd-backup-local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: Immediate ``` 2. Apply the `StorageClass` CR by running the following command: ```terminal $ oc apply -f etcd-backup-local-storage.yaml ``` 3. Create a PV named `etcd-backup-pv-fs.yaml` from the applied `StorageClass` with contents such as the following example: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: etcd-backup-pv-fs spec: capacity: storage: 100Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete storageClassName: etcd-backup-local-storage local: path: /mnt/ nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ``` where: `spec.capacity.storage` : Specifies the amount of storage available to the PV. Adjust this value for your requirements. `spec.nodeAffinity.required.nodeSelectorTerms.matchExpressions.values` : Specifies the control plane node to attach this PV to. Replace with the actual node name. > [!TIP] > List the available nodes by running the following command: > > ```terminal > $ oc get nodes > ``` 4. Verify that the PV was created by running the following command: ```terminal $ oc get pv ``` ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE etcd-backup-pv-fs 100Gi RWX Delete Available etcd-backup-local-storage 10s ``` 5. Create a PVC named `etcd-backup-pvc.yaml` with contents such as the following example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: etcd-backup-pvc spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: 10Gi storageClassName: etcd-backup-local-storage ``` where: `spec.resources.requests.storage` : Specifies the amount of storage available to the PVC. Adjust this value for your requirements. 6. Apply the PVC by running the following command: ```terminal $ oc apply -f etcd-backup-pvc.yaml ``` 3. Create a CR file named `etcd-recurring-backups.yaml`. The contents of the CR define the schedule and retention type of automated backups. - For the default retention type of `RetentionNumber` with 15 retained backups, use contents such as the following example: ```yaml apiVersion: config.openshift.io/v1alpha1 kind: Backup metadata: name: etcd-recurring-backup spec: etcd: schedule: "20 4 * * *" timeZone: "UTC" pvcName: etcd-backup-pvc ``` where: `spec.etcd.schedule` : Specifies the `CronTab` schedule for recurring backups. Adjust this value for your needs. 4. To use retention based on the maximum number of backups, add the following key-value pairs to the `etcd` key: ```yaml spec: etcd: retentionPolicy: retentionType: RetentionNumber retentionNumber: maxNumberOfBackups: 5 ``` where: `spec.etcd.retentionPolicy.retentionType` : Specifies the retention type. Defaults to `RetentionNumber` if unspecified. `spec.etcd.retentionPolicy.retentionNumber.maxNumberOfBackups` : Specifies the maximum number of backups to retain. Adjust this value for your needs. Defaults to 15 backups if unspecified. > [!WARNING] > A known issue causes the number of retained backups to be one greater than the configured value. 5. For retention based on the file size of backups, use the following: ```yaml spec: etcd: retentionPolicy: retentionType: RetentionSize retentionSize: maxSizeOfBackupsGb: 20 ``` where: `spec.etcd.retentionPolicy.retentionSize.maxSizeOfBackupsGb` : Specifies the maximum file size of the retained backups in gigabytes. Adjust this value for your needs. Defaults to 10 GB if unspecified. > [!WARNING] > A known issue causes the maximum size of retained backups to be up to 10 GB greater than the configured value. 6. Create the cron job defined by the CRD by running the following command: ```terminal $ oc create -f etcd-recurring-backup.yaml ``` 7. To find the created cron job, run the following command: ```terminal $ oc get cronjob -n openshift-etcd ```