--- title: CSI volume snapshots --- # CSI volume snapshots {#persistent-storage-csi-snapshots} Container Storage Interface (CSI) snapshots capture point-in-time copies for data protection and recovery. Snapshots enable restoring volumes to previous states or creating new volumes from existing data using `VolumeSnapshot`, `VolumeSnapshotContent`, and `VolumeSnapshotClass` objects. ## Overview of CSI volume snapshots {#persistent-storage-csi-snapshots-overview_persistent-storage-csi-snapshots} A snapshot represents the state of the storage volume in a cluster at a particular point in time. Volume snapshots can be used to provision a new volume. OpenShift Container Platform supports Container Storage Interface (CSI) volume snapshots by default. However, a specific CSI driver is required. Familiarity with persistent volumes is suggested. For information about persistent volumes, see "Persistent volumes". With CSI volume snapshots, a cluster administrator can: - Deploy a third-party CSI driver that supports snapshots. - Create a new persistent volume claim (PVC) from an existing volume snapshot. - Take a snapshot of an existing PVC. - Restore a snapshot as a different PVC. - Delete an existing volume snapshot. With CSI volume snapshots, an app developer can: - Use volume snapshots as building blocks for developing application- or cluster-level storage backup solutions. - Rapidly rollback to a previous development version. - Use storage more efficiently by not having to make a full copy each time. Be aware of the following when using volume snapshots: - Support is only available for CSI drivers. In-tree and FlexVolumes are not supported. - OpenShift Container Platform only includes select CSI drivers. For CSI drivers that are not provided by an OpenShift Container Platform Driver Operator, it is recommended to use the CSI drivers provided by the Kubernetes community or known storage vendors. Follow the installation instructions furnished by the CSI driver provider. For information about Kubernetes CSI drivers, see "Kubernetes CSI Developer Documentation". - CSI drivers may or may not have implemented the volume snapshot functionality. CSI drivers that have provided support for volume snapshots will likely use the `csi-external-snapshotter` sidecar. See documentation provided by the CSI driver for details. ## CSI snapshot controller and sidecar {#persistent-storage-csi-snapshots-controller-sidecar_persistent-storage-csi-snapshots} Container Storage Interface (CSI) snapshots require two components: a controller deployed by OpenShift Container Platform to the control plane, and a vendor-provided sidecar with the CSI driver. The controller manages `VolumeSnapshot` bindings while the sidecar triggers create and delete operations. The CSI snapshot controller and sidecar provide volume snapshotting through the OpenShift Container Platform API. These external components run in the cluster. The external controller is deployed by the CSI Snapshot Controller Operator. External controller : The CSI snapshot controller binds `VolumeSnapshot` and `VolumeSnapshotContent` objects. The controller manages dynamic provisioning by creating and deleting `VolumeSnapshotContent` objects. External sidecar : Your CSI driver vendor provides the `csi-external-snapshotter` sidecar. This is a separate helper container that is deployed with the CSI driver. The sidecar manages snapshots by triggering `CreateSnapshot` and `DeleteSnapshot` operations. Follow the installation instructions provided by your vendor. ## About the CSI Snapshot Controller Operator {#persistent-storage-csi-snapshots-operator_persistent-storage-csi-snapshots} You can manage volume snapshots in OpenShift Container Platform by using the CSI Snapshot Controller Operator’s custom resource definitions (CRDs) for snapshot requests, storage, and configuration. The Container Storage Interface (CSI) Snapshot Controller Operator runs in the `openshift-cluster-storage-operator` namespace. It is installed by the Cluster Version Operator (CVO) in all clusters by default. The CSI Snapshot Controller Operator installs the CSI snapshot controller, which runs in the `openshift-cluster-storage-operator` namespace. ### Volume snapshot CRDs {#_volume_snapshot_crds} During OpenShift Container Platform installation, the CSI Snapshot Controller Operator creates the following snapshot custom resource definitions (CRDs) in the `snapshot.storage.k8s.io/v1` API group: `VolumeSnapshotContent` : A snapshot taken of a volume in the cluster that has been provisioned by a cluster administrator. Similar to the `PersistentVolume` object, the `VolumeSnapshotContent` CRD is a cluster resource that points to a real snapshot in the storage back end. For manually pre-provisioned snapshots, a cluster administrator creates a number of `VolumeSnapshotContent` CRDs. These carry the details of the real volume snapshot in the storage system. The `VolumeSnapshotContent` CRD is not namespaced and is for use by a cluster administrator. `VolumeSnapshot` : Similar to the `PersistentVolumeClaim` object, the `VolumeSnapshot` CRD defines a developer request for a snapshot. The CSI Snapshot Controller Operator runs the CSI snapshot controller, which handles the binding of a `VolumeSnapshot` CRD with an appropriate `VolumeSnapshotContent` CRD. The binding is a one-to-one mapping. The `VolumeSnapshot` CRD is namespaced. A developer uses the CRD as a distinct request for a snapshot. `VolumeSnapshotClass` : The `VolumeSnapshotClass` CRD allows a cluster administrator to specify different attributes belonging to a `VolumeSnapshot` object. These attributes may differ among snapshots taken of the same volume on the storage system, in which case they would not be expressed by using the same storage class of a persistent volume claim. The `VolumeSnapshotClass` CRD defines the parameters for the `csi-external-snapshotter` sidecar to use when creating a snapshot. This allows the storage back end to know what kind of snapshot to dynamically create if multiple options are supported. Dynamically provisioned snapshots use the `VolumeSnapshotClass` CRD to specify storage-provider-specific parameters to use when creating a snapshot. The `VolumeSnapshotContentClass` CRD is not namespaced and is for use by a cluster administrator to enable global configuration options for their storage back end. For Google Cloud Platform (GCP) persistent disk (PD) storage CSI, there is a non-default `VolumeSnapshotClass`, named `csi-gce-pd-vsc-images`, that uses the `snapshot-type`: `images` parameter. When using KubeVirt, this allows you to create VMs from "golden images" (templates saved as snapshots). If you want to use the images volume snapshot class for dynamic snapshot provisioning, you can either: - Make the images volume snapshot class the default by changing the `snapshot.storage.kubernetes.io/is-default-class` annotation to `true`. Also, for the normal default volume snapshot class, `csi-gce-pd-vsc`, be sure to change this parameter to `false`. - When creating the snapshot object, be sure to set `volumeSnapshotClassName` to `csi-gce-pd-vsc-images`. For information about creating volume snapshots, see "Dynamically creating a volume snapshot" and "Statically creating a volume snapshot". ```yaml {title="Example images volume snapshot class YAML file"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-gce-pd-vsc-images driver: pd.csi.storage.gke.io parameters: snapshot-type: images ``` - `metadata.name:csi-gce-pd-vsc-images`: Specifies the name for the non-default images volume snapshot class. - `parameters: snapshot-type: images`: Defines the snapshot as a "golden image" or a bootable template, rather than the standard disk backup. ## Volume snapshot provisioning {#persistent-storage-csi-snapshots-provision_persistent-storage-csi-snapshots} You can provision volume snapshots in OpenShift Container Platform using either dynamic provisioning to create new snapshots on-demand or static provisioning to reference pre-existing snapshots. ### Dynamic provisioning {#snapshots-dynamic-provisioning_persistent-storage-csi-snapshots} Instead of using a preexisting snapshot, you can request that a snapshot be taken dynamically from a persistent volume claim. Parameters are specified using a `VolumeSnapshotClass` CRD. ### Static provisioning {#snapshots-manual-provisioning_persistent-storage-csi-snapshots} As a cluster administrator, you can manually pre-provision a number of `VolumeSnapshotContent` objects. These carry the real volume snapshot details available to cluster users. ## Dynamically creating a volume snapshot {#persistent-storage-csi-snapshots-create_persistent-storage-csi-snapshots} To create a point-in-time backup of a persistent volume claim (PVC), dynamically provision a volume snapshot by defining a VolumeSnapshotClass and VolumeSnapshot that automate the snapshot creation. When you create a `VolumeSnapshot` object, OpenShift Container Platform creates a volume snapshot. **Prerequisites** - Logged in to a running OpenShift Container Platform cluster. - A PVC created using a CSI driver that supports `VolumeSnapshot` objects. - A storage class to provision the storage back end. - No pods are using the persistent volume claim (PVC) that you want to take a snapshot of. > [!WARNING] > Creating a volume snapshot of a PVC that is in use by a pod can cause unwritten data and cached data to be excluded from the snapshot. To ensure that all data is written to the disk, delete the pod that is using the PVC before creating the snapshot. **Procedure** 1. Create a file with the `VolumeSnapshotClass` object described by the following YAML: ```yaml {title="Example volumesnapshotclass.yaml"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-hostpath-snap driver: hostpath.csi.k8s.io deletionPolicy: Delete ``` `driver` is the name of the CSI driver that is used to create snapshots of this `VolumeSnapshotClass` object. The name must be the same as the `Provisioner` field of the storage class that is responsible for the PVC that is being snapshotted. > [!NOTE] > Depending on the driver that you used to configure persistent storage, additional parameters might be required. You can also use an existing `VolumeSnapshotClass` object. 2. Create the object you saved in the previous step by entering the following command: ```terminal $ oc create -f volumesnapshotclass.yaml ``` 3. Create a `VolumeSnapshot` object: ```yaml {title="Example volumesnapshot-dynamic.yaml"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysnap spec: volumeSnapshotClassName: csi-hostpath-snap source: persistentVolumeClaimName: myclaim ``` - `spec.volumeSnapshotClassName`: Specifies the request for a particular class by the volume snapshot. If the `volumeSnapshotClassName` setting is absent and there is a default volume snapshot class, a snapshot is created with the default volume snapshot class name. But if the field is absent and no default volume snapshot class exists, then no snapshot is created. - `spec.source.persistentVolumeClaimName`: Specifies the name of the `PersistentVolumeClaim` object bound to a persistent volume. This defines what you want to create a snapshot of. Required for dynamically provisioning a snapshot. 4. Create the object you saved in the previous step by entering the following command: ```terminal $ oc create -f volumesnapshot-dynamic.yaml ``` **Verification** 1. After the snapshot has been created in the cluster, additional details about the snapshot are available. To display details about the volume snapshot that was created, run the following command: ```terminal $ oc describe volumesnapshot mysnap ``` The following example displays details about the `mysnap` volume snapshot: ```yaml {title="Example volumesnapshot.yaml"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysnap spec: source: persistentVolumeClaimName: myclaim volumeSnapshotClassName: csi-hostpath-snap status: boundVolumeSnapshotContentName: snapcontent-1af4989e-a365-4286-96f8-d5dcd65d78d6 creationTime: "2020-01-29T12:24:30Z" readyToUse: true restoreSize: 500Mi ``` - `status.boundVolumeSnapshotContentName`: This parameter is the pointer to the actual storage content that was created by the controller. - `status.creationTime`: Specifies the time when the snapshot was created. The snapshot contains the volume content that was available at this indicated time. - `status.readyToUse`: Specifies the readiness of the snapshot. If the value is set to `true`, the snapshot can be used to restore as a new PVC. If the value is set to `false`, the snapshot was created. However, the storage back end needs to perform additional tasks to make the snapshot usable so that it can be restored as a new volume. For example, Amazon Elastic Block Store data might be moved to a different, less expensive location, which can take several minutes. 2. To verify that the volume snapshot was created, enter the following command: ```terminal $ oc get volumesnapshotcontent ``` The pointer to the actual content is displayed. If the `boundVolumeSnapshotContentName` field is populated, a `VolumeSnapshotContent` object exists and the snapshot was created. 3. To verify that the snapshot is ready, confirm that the `VolumeSnapshot` object has `readyToUse: true`. ## Statically creating a volume snapshot {#persistent-storage-csi-snapshots-create-static_persistent-storage-csi-snapshots} To make a pre-existing storage snapshot available in OpenShift Container Platform, manually create a volume snapshot that references the existing snapshot content by name. **Prerequisites** - Logged in to a running OpenShift Container Platform cluster. - A PVC created using a CSI driver that supports `VolumeSnapshot` objects. - A storage class to provision the storage back end. - No pods are using the persistent volume claim (PVC) that you want to take a snapshot of. > [!WARNING] > Creating a volume snapshot of a PVC that is in use by a pod can cause unwritten data and cached data to be excluded from the snapshot. To ensure that all data is written to the disk, delete the pod that is using the PVC before creating the snapshot. **Procedure** 1. Create a file with the `VolumeSnapshotClass` object described by the following YAML: ```yaml {title="Example volumesnapshotclass.yaml"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-hostpath-snap driver: hostpath.csi.k8s.io deletionPolicy: Delete ``` `driver` is the name of the CSI driver that is used to create snapshots of this `VolumeSnapshotClass` object. The name must be the same as the `Provisioner` field of the storage class that is responsible for the PVC that is being snapshotted. > [!NOTE] > Depending on the driver that you used to configure persistent storage, additional parameters might be required. You can also use an existing `VolumeSnapshotClass` object. 2. Create the object you saved in the previous step by entering the following command: ```terminal $ oc create -f volumesnapshotclass.yaml ``` 3. Provide a value for the `volumeSnapshotContentName` parameter as the source for the snapshot: ```yaml {title="Example volumesnapshot-manual.yaml"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: snapshot-demo spec: source: volumeSnapshotContentName: mycontent ``` `spec.source.volumeSnapshotContentName` is required for pre-provisioned snapshots. 4. Create the object you saved in the previous step by entering the following command: ```terminal $ oc create -f volumesnapshot-manual.yaml ``` **Verification** After the snapshot has been created in the cluster, additional details about the snapshot are available. 1. To display details about the volume snapshot that was created, enter the following command: ```terminal $ oc describe volumesnapshot mysnap ``` The following example displays details about the `mysnap` volume snapshot: ```yaml {title="Example volumesnapshot.yaml"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysnap spec: source: persistentVolumeClaimName: myclaim volumeSnapshotClassName: csi-hostpath-snap status: boundVolumeSnapshotContentName: snapcontent-1af4989e-a365-4286-96f8-d5dcd65d78d6 creationTime: "2020-01-29T12:24:30Z" readyToUse: true restoreSize: 500Mi ``` - `status.boundVolumeSnapshotContentName`: Specifies the pointer to the actual storage content that was created by the controller. - `status.creationTime`: Specifies the time when the snapshot was created. The snapshot contains the volume content that was available at this indicated time. - `status.readyToUse`: If the value is set to `true`, the snapshot can be used to restore as a new PVC. If the value is set to `false`, the snapshot was created. However, the storage back end needs to perform additional tasks to make the snapshot usable so that it can be restored as a new volume. For example, Amazon Elastic Block Store data might be moved to a different, less expensive location, which can take several minutes. 2. To verify that the volume snapshot was created, enter the following command: ```terminal $ oc get volumesnapshotcontent ``` The pointer to the actual content is displayed. If the `boundVolumeSnapshotContentName` field is populated, a `VolumeSnapshotContent` object exists and the snapshot was created. 3. To verify that the snapshot is ready, confirm that the `VolumeSnapshot` object has `readyToUse: true`. ## Deleting a volume snapshot {#persistent-storage-csi-snapshots-delete_persistent-storage-csi-snapshots} To clean up unneeded snapshots and free storage resources, delete volume snapshots by setting a deletion policy that controls whether the underlying content is retained or removed. **Procedure** 1. Specify the deletion policy that you require in the `VolumeSnapshotClass` object, as shown in the following example: ```yaml {title="Example volumesnapshotclass.yaml file"} apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-hostpath-snap driver: hostpath.csi.k8s.io deletionPolicy: Delete # ... ``` When deleting the volume snapshot, if `deletionPolicy` is set to `Delete`, the underlying snapshot is deleted along with the `VolumeSnapshotContent` object. If the `Retain` value is set, both the underlying snapshot and `VolumeSnapshotContent` object remain. > [!NOTE] > If the `Retain` value is set and the `VolumeSnapshot` object is deleted without deleting the corresponding `VolumeSnapshotContent` object, the content remains. The snapshot itself is also retained in the storage back end. 2. Delete the volume snapshot by entering the following command: ```terminal $ oc delete volumesnapshot ``` Replace `` with the name of the volume snapshot you want to delete. ```terminal {title="Example output"} volumesnapshot.snapshot.storage.k8s.io "mysnapshot" deleted ``` 3. If the deletion policy is set to `Retain`, delete the volume snapshot content by entering the following command: ```terminal $ oc delete volumesnapshotcontent ``` Replace `` with the content you want to delete. 4. Optional: If the `VolumeSnapshot` object is not successfully deleted, enter the following command to remove any finalizers for the leftover resource so that the delete operation can continue: > [!IMPORTANT] > Only remove the finalizers if you are confident that there are no existing references from either persistent volume claims or volume snapshot contents to the `VolumeSnapshot` object. Even with the `--force` option, the delete operation does not delete snapshot objects until all finalizers are removed. ```terminal $ oc patch -n $PROJECT volumesnapshot/$NAME --type=merge -p '{"metadata": {"finalizers":null}}' ``` ```terminal {title="Example output"} volumesnapshotclass.snapshot.storage.k8s.io "csi-ocs-rbd-snapclass" deleted ``` The finalizers are removed and the volume snapshot is deleted. ## Restoring a volume snapshot {#persistent-storage-csi-snapshots-restore_persistent-storage-csi-snapshots} To recover previous data or reuse snapshot data, create a new persistent volume claim (PVC) that is pre-populated with content from an existing volume snapshot. The `VolumeSnapshot` CRD content can be used to restore the existing volume to a previous state. After your `VolumeSnapshot` CRD is bound and the `readyToUse` value is set to `true`, you can use that resource to provision a new volume that is pre-populated with data from the snapshot. **Prerequisites** - Logged in to a running OpenShift Container Platform cluster. - A persistent volume claim (PVC) created using a Container Storage Interface (CSI) driver that supports volume snapshots. - A storage class to provision the storage back end. - A volume snapshot has been created and is ready to use. **Procedure** 1. Specify a `VolumeSnapshot` data source on a PVC as shown in the following: ```yaml {title="pvc-restore.yaml"} apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim-restore spec: storageClassName: csi-hostpath-sc dataSource: name: mysnap kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` - `spec.dataSource.name`: Specifies the name of the `VolumeSnapshot` object representing the snapshot to use as source. - `spec.dataSource.kind`: Must be set to the `VolumeSnapshot` value. - `spec.dataSource.apiGroup`: Must be set to the `snapshot.storage.k8s.io` value. 2. Create a PVC by entering the following command: ```terminal $ oc create -f pvc-restore.yaml ``` 3. Verify that the restored PVC has been created by entering the following command: ```terminal $ oc get pvc ``` A new PVC such as `myclaim-restore` is displayed. ## Changing the maximum number of snapshots for vSphere {#vsphere-change-max-snapshot_persistent-storage-csi-snapshots} Configure the maximum number of snapshots per volume globally or for specific datastore types to balance storage capacity and performance in your vSphere environment. The default maximum number of snapshots per volume in vSphere Container Storage Interface (CSI) is 3. You can change the maximum number up to 32 per volume. However, be aware that increasing the snapshot maximum involves a performance trade off, so for better performance use only 2 to 3 snapshots per volume. For more VMware snapshot performance recommendations, see "Best practices for using VMware snapshots in the vSphere environment". **Prerequisites** - Access to the cluster with administrator rights. **Procedure** 1. Check the current secret by the running the following command: ```terminal $ oc -n openshift-cluster-csi-drivers get secret/vsphere-csi-config-secret -o jsonpath='{.data.cloud\.conf}' | base64 -d ``` ```terminal {title="Example output"} # Labels with topology values are added dynamically via operator [Global] cluster-id = vsphere-01-cwv8p # Populate VCenters (multi) after here [VirtualCenter "vcenter.openshift.com"] insecure-flag = true datacenters = DEVQEdatacenter password = "xxxxxxxx" user = "xxxxxxxx@devcluster.openshift.com" migration-datastore-url = ds:///vmfs/volumes/vsan:52c842f232751e0d-3253aadeac21ca82/ ``` In this example, the global maximum number of snapshots is not configured, so the default value of 3 is applied. 2. Change the snapshot limit by running the following command: - Set **global** snapshot limit: ```terminal $ oc patch clustercsidriver/csi.vsphere.vmware.com --type=merge -p '{"spec":{"driverConfig":{"vSphere":{"globalMaxSnapshotsPerBlockVolume": 10}}}}' clustercsidriver.operator.openshift.io/csi.vsphere.vmware.com patched ``` In this example, the global limit is being changed to 10 (`globalMaxSnapshotsPerBlockVolume` set to 10). - Set **Virtual Volume** snapshot limit: This parameter sets the limit on the Virtual Volumes datastore only. The Virtual Volume maximum snapshot limit overrides the global constraint if set, but defaults to the global limit if it is not set. ```terminal $ oc patch clustercsidriver/csi.vsphere.vmware.com --type=merge -p '{"spec":{"driverConfig":{"vSphere":{"granularMaxSnapshotsPerBlockVolumeInVVOL": 5}}}}' clustercsidriver.operator.openshift.io/csi.vsphere.vmware.com patched ``` In this example, the Virtual Volume limit is being changed to 5 (`granularMaxSnapshotsPerBlockVolumeInVVOL` set to 5). - Set **vSAN** snapshot limit: This parameter sets the limit on the vSAN datastore only. The vSAN maximum snapshot limit overrides the global constraint if set, but defaults to the global limit if it is not set. You can set a maximum value of 32 under vSAN ESA setup. ```terminal $ oc patch clustercsidriver/csi.vsphere.vmware.com --type=merge -p '{"spec":{"driverConfig":{"vSphere":{"granularMaxSnapshotsPerBlockVolumeInVSAN": 7}}}}' clustercsidriver.operator.openshift.io/csi.vsphere.vmware.com patched ``` In this example, the vSAN limit is being changed to 7 (`granularMaxSnapshotsPerBlockVolumeInVSAN` set to 7). **Verification** - Verify that any changes you made are reflected in the config map by running the following command: ```terminal $ oc -n openshift-cluster-csi-drivers get secret/vsphere-csi-config-secret -o jsonpath='{.data.cloud\.conf}' | base64 -d ``` ```terminal {title="Example output"} # Labels with topology values are added dynamically via operator [Global] cluster-id = vsphere-01-cwv8p # Populate VCenters (multi) after here [VirtualCenter "vcenter.openshift.com"] insecure-flag = true datacenters = DEVQEdatacenter password = "xxxxxxxx" user = "xxxxxxxx@devcluster.openshift.com" migration-datastore-url = ds:///vmfs/volumes/vsan:52c842f232751e0d-3253aadeac21ca82/ [Snapshot] global-max-snapshots-per-block-volume = 10 ``` The parameter `global-max-snapshots-per-block-volume` is now set to 10. **Additional resources** {._additional-resources} - [Persistent volumes](/openshift-docs-markdown/storage/understanding-persistent-storage#persistent-volumes_understanding-persistent-storage) - [Kubernetes CSI Developer Documentation](https://kubernetes-csi.github.io/docs/drivers.html) - [Dynamically creating a volume snapshot](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi-snapshots#persistent-storage-csi-snapshots-create_persistent-storage-csi-snapshots) - [Statically creating a volume snapshot](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi-snapshots#persistent-storage-csi-snapshots-create-static_persistent-storage-csi-snapshots) - [Best practices for using VMware snapshots in the vSphere environment](https://kb.vmware.com/s/article/1025279)