--- title: Persistent storage using local volumes --- # Persistent storage using local volumes {#persistent-storage-using-local-volume} OpenShift Container Platform can be provisioned with persistent storage by using local volumes. Local persistent volumes allow you to access local storage devices, such as a disk or partition, by using the standard persistent volume claim interface. ## Local Storage Operator overview {#local-storage-overview_persistent-storage-local} Local volumes can be used without manually scheduling pods to nodes because the system is aware of the volume node constraints. However, local volumes are still subject to the availability of the underlying node and are not suitable for all applications. > [!NOTE] > Local volumes can only be used as a statically created persistent volume. ## Installing the Local Storage Operator {#local-storage-install-overview_persistent-storage-local} You can install the Local Storage Operator (LSO) to provision and manage local persistent storage volumes in your cluster. 1. Create the `openshift-local-storage` project. 2. Install and configure the LSO by using either the CLI or OpenShift Container Platform web console. ### Creating a local project for Local Storage Operator {#local-storage-install_persistent-storage-local} To install the Local Storage Operator (LSO) to provision and manage local persistent storage volumes in your cluster, first create the `openshift-local-storage` project. **Prerequisites** - Access to the OpenShift Container Platform web console or command-line interface (CLI). **Procedure** 1. Create the `openshift-local-storage` project by running the following command: ```terminal $ oc adm new-project openshift-local-storage ``` 2. Optional: Allow local storage creation on infrastructure nodes. You might want to use the LSO to create volumes on infrastructure nodes in support of components such as logging and monitoring. You must adjust the default node selector so that the LSO includes the infrastructure nodes, and not just worker nodes. To block the LSO from inheriting the cluster-wide default selector, run the following command: ```terminal $ oc annotate namespace openshift-local-storage openshift.io/node-selector='' ``` 3. Optional: Allow local storage to run on the management pool of CPUs in single-node deployment. Use the LSO in single-node deployments and allow the use of CPUs that belong to the `management` pool. Perform this step on single-node installations that use management workload partitioning. To allow LSO to run on the management CPU pool, run following command: ```terminal $ oc annotate namespace openshift-local-storage workload.openshift.io/allowed='management' ``` **Next steps** Install the LSO Operator. ### Installing the Local Storage Operator by using the CLI {#local-storage-install-cli_persistent-storage-local} Install the Local Storage Operator (LSO) to provision and manage local persistent storage volumes in your cluster using the command-line interface (CLI). The LSO is not installed in OpenShift Container Platform by default. Use the following procedure to install and configure this Operator to enable local volumes in your cluster. **Prerequisites** - Access to an `openshift-local-storage` project. - Access to the OpenShift Container Platform command-line interface (CLI). **Procedure** 1. Create an object YAML file to define an Operator group and subscription for the LSO, such as `openshift-local-storage.yaml`: ```yaml {title="Example openshift-local-storage.yaml"} apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: local-operator-group namespace: openshift-local-storage spec: targetNamespaces: - openshift-local-storage --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: local-storage-operator namespace: openshift-local-storage spec: channel: stable installPlanApproval: Automatic name: local-storage-operator source: redhat-operators sourceNamespace: openshift-marketplace ``` `spec.installPlanApproval` is the user approval policy for an install plan. 2. Create the LSO object by running the following command: ```terminal $ oc apply -f openshift-local-storage.yaml ``` The Operator Lifecycle Manager (OLM) is now aware of the LSO. A ClusterServiceVersion (CSV) for the Operator appears in the target namespace, and APIs provided by the Operator should be available for creation. 3. Verify local storage installation by checking that all pods and the LSO have been created: 1. Check that all the required pods have been created by running the following command: ```terminal $ oc -n openshift-local-storage get pods ``` ```terminal {title="Example output"} NAME READY STATUS RESTARTS AGE local-storage-operator-746bf599c9-vlt5t 1/1 Running 0 19m ``` 2. Check the ClusterServiceVersion (CSV) YAML manifest to see that the LSO is available in the `openshift-local-storage` project: ```terminal $ oc get csvs -n openshift-local-storage ``` ```terminal {title="Example output"} NAME DISPLAY VERSION REPLACES PHASE local-storage-operator.4.2.26-202003230335 Local Storage 4.2.26-202003230335 Succeeded ``` **Result** After all checks have passed, the LSO is installed successfully. ### Installing the Local Storage Operator by using the OpenShift Container Platform web console {#local-storage-install-console_persistent-storage-local} Install the Local Storage Operator (LSO) to provision and manage local persistent storage volumes in your cluster using the OpenShift Container Platform web console. The LSO is not installed in OpenShift Container Platform by default. Use the following procedure to install and configure this Operator to enable local volumes in your cluster. **Prerequisites** - Access to an `openshift-local-storage` project. - Access to the OpenShift Container Platform web console. **Procedure** 1. Log in to the OpenShift Container Platform web console. 2. Navigate to **Ecosystem** → **Software Catalog**. 3. Type **Local Storage** into the filter box to locate the LSO. 4. Click **Install**. 5. On the **Install Operator** page, select **A specific namespace on the cluster**. Select **openshift-local-storage** from the drop-down menu. 6. Adjust the values for **Update Channel** and **Approval Strategy** to the values that you want. 7. Click **Install**. **Result** After finishing, the LSO is listed in the **Installed Operators** section of the OpenShift Container Platform web console. ## Provisioning local volumes by using the Local Storage Operator {#local-volume-cr_persistent-storage-local} Provision local volumes for persistent storage by creating a `LocalVolume` resource that defines nodes and device paths, because local volumes cannot be created by dynamic provisioning and must be statically provisioned by the Local Storage Operator (LSO). **Prerequisites** - The LSO is installed. - You have a local disk that meets the following conditions: - It is attached to a node. - It is not mounted. - It does not contain partitions. **Procedure** 1. Create the local volume resource. This resource must define the nodes and paths to the local volumes. > [!IMPORTANT] > - Editing the `LocalVolume` object does not change the `fsType` or `volumeMode` of existing persistent volumes because doing so might result in a destructive operation. > - Do not use different storage class names for the same device. Doing so creates multiple persistent volumes (PVs). ```yaml {title="Example: Filesystem"} apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "local-disks" namespace: "openshift-local-storage" spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ip-10-0-140-183 - ip-10-0-158-139 - ip-10-0-164-33 storageClassDevices: - storageClassName: "local-sc" forceWipeDevicesAndDestroyAllData: false volumeMode: Filesystem fsType: xfs devicePaths: - /path/to/device ``` - `metadata.namespace`: Specifies the namespace where the LSO is installed. - `spec.nodeSelector`: (Optional) A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from `oc get node`. If a value is not defined, then the LSO will attempt to find matching disks on all available nodes. - `spec.storageClassDevices.storageClassName`: Specifies the name of the storage class to use when creating persistent volume objects. The LSO automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes. - `spec.storageClassDevices.forceWipeDevicesAndDestroyAllData`: This setting defines whether or not to call `wipefs`, which removes partition table signatures (magic strings) making the disk ready to use for LSO provisioning. No other data besides signatures is erased. The default is "false" (`wipefs` is not invoked). Setting `forceWipeDevicesAndDestroyAllData` to "true" can be useful in scenarios where previous data can remain on disks that need to be re-used. In these scenarios, setting this field to true eliminates the need for administrators to erase the disks manually. Such cases can include single-node OpenShift cluster environments where a node can be redeployed multiple times or when using OpenShift Data Foundation, where previous data can remain on the disks planned to be consumed as object storage devices (OSDs). - `spec.storageClassDevices.volumeMode`: Specifies the volume mode, either `Filesystem` or `Block`, that defines the type of local volumes. > [!NOTE] > A raw block volume (`volumeMode: Block`) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices. - `spec.storageClassDevices.fsType`: The file system that is created when the local volume is mounted for the first time. - `spec.storageClassDevices.devicePaths`: The path containing a list of local storage devices to choose from. Replace this value with your actual local disks filepath to the `LocalVolume` resource `by-id`, such as `/dev/disk/by-id/wwn`. PVs are created for these local disks when the provisioner is deployed successfully. > [!NOTE] > If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the `virsh edit ` command to add the `mydisk` definition. ```yaml {title="Example: Block"} apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "local-disks" namespace: "openshift-local-storage" spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ip-10-0-136-143 - ip-10-0-140-255 - ip-10-0-144-180 storageClassDevices: - storageClassName: "local-sc" forceWipeDevicesAndDestroyAllData: false volumeMode: Block devicePaths: - /path/to/device ``` - `metadata.namespace`: Specifies the namespace where the LSO is installed. - `spec.nodeSelector`: (Optional) A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from `oc get node`. If a value is not defined, then the LSO will attempt to find matching disks on all available nodes. - `spec.storageClassDevices.storageClassName`: Specifies the name of the storage class to use when creating persistent volume objects. - `spec.storageClassDevices.forceWipeDevicesAndDestroyAllData`: This setting defines whether or not to call `wipefs`, which removes partition table signatures (magic strings) making the disk ready to use for LSO provisioning. No other data besides signatures is erased. The default is "false" (`wipefs` is not invoked). Setting `forceWipeDevicesAndDestroyAllData` to "true" can be useful in scenarios where previous data can remain on disks that need to be re-used. In these scenarios, setting this field to true eliminates the need for administrators to erase the disks manually. Such cases can include single-node OpenShift cluster environments where a node can be redeployed multiple times or when using OpenShift Data Foundation, where previous data can remain on the disks planned to be consumed as object storage devices (OSDs). - `spec.storageClassDevices.volumeMode`: Specifies the volume mode, either `Filesystem` or `Block`, that defines the type of local volumes. - `spec.storageClassDevices.devicePaths`: Specifies the path containing a list of local storage devices to choose from. Replace this value with your actual local disks filepath to the `LocalVolume` resource `by-id`, such as `dev/disk/by-id/wwn`. PVs are created for these local disks when the provisioner is deployed successfully. > [!NOTE] > If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the `virsh edit ` command to add the `mydisk` definition. 2. Create the local volume resource in your OpenShift Container Platform cluster. Specify the file you just created: ```terminal $ oc create -f .yaml ``` 3. Verify that the provisioner was created and that the corresponding daemon sets were created: ```terminal $ oc get all -n openshift-local-storage ``` ```terminal {title="Example output"} NAME READY STATUS RESTARTS AGE pod/diskmaker-manager-9wzms 1/1 Running 0 5m43s pod/diskmaker-manager-jgvjp 1/1 Running 0 5m43s pod/diskmaker-manager-tbdsj 1/1 Running 0 5m43s pod/local-storage-operator-7db4bd9f79-t6k87 1/1 Running 0 14m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/local-storage-operator-metrics ClusterIP 172.30.135.36 8383/TCP,8686/TCP 14m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/diskmaker-manager 3 3 3 3 3 5m43s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/local-storage-operator 1/1 1 1 14m NAME DESIRED CURRENT READY AGE replicaset.apps/local-storage-operator-7db4bd9f79 1 1 1 14m ``` Note the desired and current number of daemon set processes. A desired count of `0` indicates that the label selectors were invalid. 4. Verify that the persistent volumes were created by running the following command: ```terminal $ oc get pv ``` ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-1cec77cf 100Gi RWO Delete Available local-sc 88m local-pv-2ef7cd2a 100Gi RWO Delete Available local-sc 82m local-pv-3fa1c73 100Gi RWO Delete Available local-sc 48m ``` ## Provisioning local volumes without the Local Storage Operator {#local-create-cr-manual_persistent-storage-local} Provision local volumes manually by defining `PersistentVolume` objects without using the Local Storage Operator (LSO), though this approach includes risk of potential data leaks and the operator is recommended for automating device lifecycle. > [!IMPORTANT] > Manual provisioning of persistent volumes (PVs) includes the risk of potential data leaks across PV reuse when persistent volume claims (PVCs) are deleted. The Local Storage Operator is recommended for automating the life cycle of devices when provisioning local PVs. **Prerequisites** - Local disks are attached to the OpenShift Container Platform nodes. **Procedure** 1. Define the PV. Create a file, such as `example-pv-filesystem.yaml` or `example-pv-block.yaml`, with the `PersistentVolume` object definition. This resource must define the nodes and paths to the local volumes. > [!NOTE] > Do not use different storage class names for the same device. Doing so creates multiple PVs. ```yaml {title="Example-pv-filesystem.yaml"} apiVersion: v1 kind: PersistentVolume metadata: name: example-pv-filesystem spec: capacity: storage: 100Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-sc local: path: /dev/xvdf nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - example-node ``` - `spec.volumeMode`: Specifies the volume mode, either `Filesystem` or `Block`, that defines the type of PVs. - `spec.storageClassName`: Specifies the name of the storage class to use when creating PV resources. Use a storage class that uniquely identifies this set of PVs. - `spec.local.path`: Specifies the path containing a list of local storage devices to choose from, or a directory. You can only specify a directory with `Filesystem` `volumeMode`. > [!NOTE] > A raw block volume (`volumeMode: block`) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices. ```yaml {title="Example-pv-block.yaml"} apiVersion: v1 kind: PersistentVolume metadata: name: example-pv-block spec: capacity: storage: 100Gi volumeMode: Block accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-sc local: path: /dev/xvdf nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - example-node ``` - spe`c.volumeMode`: Specifies the volume mode, either `Filesystem` or `Block`, that defines the type of PVs. - `spec.storageClassName`: Specifies the name of the storage class to use when creating PV resources. Be sure to use a storage class that uniquely identifies this set of PVs. - `spec.local.path`: Specifies the path containing a list of local storage devices to choose from. 2. Specifying the file that you just created, create the PV resource in your OpenShift Container Platform cluster by running the following command: ```terminal $ oc create -f .yaml ``` 3. Verify that the local PV was created: ```terminal $ oc get pv ``` ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE example-pv-filesystem 100Gi RWO Delete Available local-sc 3m47s example-pv1 1Gi RWO Delete Bound local-storage/pvc1 local-sc 12h example-pv2 1Gi RWO Delete Bound local-storage/pvc2 local-sc 12h example-pv3 1Gi RWO Delete Bound local-storage/pvc3 local-sc 12h ``` ## Creating the local volume persistent volume claim {#create-local-pvc_persistent-storage-local} Create a persistent volume claim (PVC) to access local volumes in your pod, because local volumes must be statically created and cannot use dynamic provisioning. **Prerequisites** - Persistent volumes have been created using the local volume provisioner. **Procedure** 1. Create the PVC using the corresponding storage class: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-pvc-name spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 100Gi storageClassName: local-sc ``` - `metadata.name`: Specifies the name of the PVC. - `spec.volumeMode`: Specifies the type of the PVC. Defaults to `Filesystem`. - `spec.resources.requests.storage`: Specifies the amount of storage available to the PVC. - `spec.storageClassName`: Specifies the name of the storage class required by the claim. 2. Create the PVC in the OpenShift Container Platform cluster, specifying the file you just created, by running the following command: ```terminal $ oc create -f .yaml ``` ## Attach the local claim {#local-pod_persistent-storage-local} After a local volume has been mapped to a persistent volume claim (PVC), attach the claim to a pod by specifying it in the pod specification to make the local storage available to the application. **Prerequisites** - A persistent volume claim exists in the same namespace. **Procedure** 1. Include the defined claim in the resource spec. The following example declares the persistent volume claim inside a pod: ```yaml apiVersion: v1 kind: Pod spec: # ... containers: volumeMounts: - name: local-disks mountPath: /data volumes: - name: local-disks persistentVolumeClaim: claimName: local-pvc-name # ... ``` - `spec...containers.volumeMounts.name`: Specifies the name of the volume to mount. - `spec...containers.volumeMounts.mountPath`: Specifies the path inside the pod where the volume is mounted. Do not mount to the container root, `/`, or any path that is the same in the host and the container. This can corrupt your host system if the container is sufficiently privileged, such as the host `/dev/pts` files. It is safe to mount the host by using `/host`. - `spec...volumes.persistentVolumeClaim.claimName`: Specifies the name of the existing persistent volume claim to use. 2. Create the resource in the OpenShift Container Platform cluster, specifying the file you just created, by running the following command: ```terminal $ oc create -f .yaml ``` ## Automating discovery and provisioning for local storage devices {#local-storage-discovery_persistent-storage-local} Automate local storage discovery and provisioning using the Local Storage Operator (LSO) to simplify installation when dynamic provisioning is not available, such as with bare metal, VMware vSphere, or Amazon Web Services (AWS) instances with attached devices. > [!IMPORTANT] > Automatic discovery and provisioning is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. > > For more information about the support scope of Red Hat Technology Preview features, see [Technology Preview Features Support Scope](https://access.redhat.com/support/offerings/techpreview/). > [!IMPORTANT] > Automatic discovery and provisioning is fully supported when used to deploy Red Hat OpenShift Data Foundation on-premise or with platform-agnostic deployment. Use the following procedure to automatically discover local devices, and to automatically provision local volumes for selected devices. > [!WARNING] > Use the `LocalVolumeSet` object with caution. When you automatically provision persistent volumes (PVs) from local disks, the local PVs might claim all devices that match. If you are using a `LocalVolumeSet` object, make sure the LSO is the only entity managing local devices on the node. Creating multiple instances of a `LocalVolumeSet` that target a node more than once is not supported. **Prerequisites** - You have cluster administrator permissions. - You have installed the LSO. - You have attached local disks to OpenShift Container Platform nodes. - You have access to the OpenShift Container Platform web console and the `oc` command-line interface (CLI). **Procedure** 1. Enable automatic discovery of local devices from the OpenShift Container Platform web console: 1. Click **Ecosystem** → **Installed Operators**. 2. In the `openshift-local-storage` namespace, click **Local Storage**. 3. Click the **Local Volume Discovery** tab. 4. Click **Create Local Volume Discovery** and then select either **Form view** or **YAML view**. 5. Configure the `LocalVolumeDiscovery` object parameters. 6. Click **Create**. The LSO creates a local volume discovery instance named `auto-discover-devices`. 2. Display a continuous list of available devices on a node: 1. Log in to the OpenShift Container Platform web console. 2. Click **Compute** → **Nodes**. 3. Click the node name that you want to open. The "Node Details" page is displayed. 4. Click the **Disks** tab to display the list of the selected devices. The device list updates continuously as local disks are added or removed. You can filter the devices by name, status, type, model, capacity, and mode. 3. Automatically provision local volumes for the discovered devices from the OpenShift Container Platform web console: 1. Click **Ecosystem** → **Installed Operators** and select **Local Storage** from the list of Operators. 2. Click **Local Volume Set** → **Create Local Volume Set**. 3. Enter a volume set name and a storage class name. 4. Click **All nodes** or **Select nodes** to apply filters accordingly. > [!NOTE] > Only worker nodes are available, regardless of whether you filter using **All nodes** or **Select nodes**. 5. Select the disk type, mode, size, and limit that you want to apply to the local volume set, and then click **Create**. A message is displayed after several minutes, indicating that the "Operator reconciled successfully." 4. Alternatively, provision local volumes for the discovered devices from the CLI: 1. Create an object YAML file to define the local volume set, such as `local-volume-set.yaml`, as shown in the following example: ```yaml {title="Example local volume set YAML file"} apiVersion: local.storage.openshift.io/v1alpha1 kind: LocalVolumeSet metadata: name: example-autodetect spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - worker-0 - worker-1 storageClassName: local-sc volumeMode: Filesystem fsType: ext4 maxDeviceCount: 10 deviceInclusionSpec: deviceTypes: - disk - part deviceMechanicalProperties: - NonRotational minSize: 10G maxSize: 100G models: - SAMSUNG - Crucial_CT525MX3 vendors: - ATA - ST2000LM ``` - `spec.storageClassName`: Determines the storage class that is created for persistent volumes that are provisioned from discovered devices. The LSO automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes. - `spec.deviceInclusionSpec.deviceTypes`: When using the local volume set feature, the LSO does not support the use of logical volume management (LVM) devices. 2. Create the local volume set object: ```terminal $ oc apply -f local-volume-set.yaml ``` 3. Verify that the local persistent volumes were dynamically provisioned based on the storage class: ```terminal $ oc get pv ``` ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-1cec77cf 100Gi RWO Delete Available local-sc 88m local-pv-2ef7cd2a 100Gi RWO Delete Available local-sc 82m local-pv-3fa1c73 100Gi RWO Delete Available local-sc 48m ``` **Next steps** Results are deleted after they are removed from the node. Symlinks must be manually removed. ## Local Storage Operator symlinks management {#local-storage-symlinks-top-level_persistent-storage-local} To prevent storage breakage during OpenShift Container Platform upgrades, OpenShift Container Platform provides a mechanism, the `LocalVolumeDeviceLink` Custom Resource Definition, to detect, alert, and remap broken symlinks without manual node-level intervention. ### Local Storage Operator symlinks overview {#local-storage-symlinks_persistent-storage-local} The Local Storage Operator (LSO) traditionally creates persistent volumes (PVs) based on `/dev/disk/by-id/` paths, following the assumption that they are stable. However, Linux kernel updates, firmware updates, or `udev` rule changes can cause these supposedly stable names to change or disappear. Administrator options : Administrators have the following notification and correction options to deal with symlink disruptions: - **Monitoring**: (default) If the current and preferred path do not match, an alert occurs, but no changes occur to the current path. - **Use existing path**: Alerts are silenced and LSO uses the existing path. - **Recreate symlinks**: Symlinks are re-created to point to the new, updated device path. ### Responding to symlinks alerts for the Local Storage Operator {#local-storage-symlinks-procedure_persistent-storage-local} To prevent storage breakage during OpenShift Container Platform upgrades, an administrator can elect to detect, alert, and remap broken symlinks without manual node-level intervention. By default, LSO engages in link monitoring and generates an alert if the current and preferred paths do not match. If an alert occurs, an administrator can choose to either have LSO: - Use the existing path. - Re-create the symlink to point to the new, updated device path. **Prerequisites** - Access to the OpenShift Container Platform web console or command-line interface (CLI) with administrative privileges. - Install the Local Storage Operator (LSO). For more information, see "Installing the Local Storage Operator". **Procedure** 1. Log in to the OpenShift Container Platform web console. 2. To view volumes generating alerts: 1. On the left navigation menu, click **Observe** > **Alerting**. 2. In the **Alert Name** filter box, search for the required LSO alerts: **Symlink alerts for LSO** | Alert | Description | | :--- | :--- | | lso_no_stable_volume_path | Device does not have a stable path and is being referenced by device name, which can change between reboots | | lso_device_link_mismatch | Device has mismatching preferred and current symlink | | lso_lv_missing_device_path | LV object has missing devicePath on actual node | Alternatively, you can list volumes triggering device link alerts by running the following command: ```terminal $ oc get localvolumedevicelink -A -o json | jq -r ' .items[] | select( .spec.policy == "None" and (.status.currentLinkTarget != .status.preferredLinkTarget or .status.currentLinkTarget == "") ) | [.metadata.namespace, .metadata.name, .spec.persistentVolumeName, .status.currentLinkTarget // "MISSING", .status.preferredLinkTarget // "NONE"] | @tsv' | column -t -s $'\t' -N "NAMESPACE,NAME,PV,CURRENT-TARGET,PREFERRED-TARGET" ``` ```terminal {title="Example output"} NAMESPACE NAME PV CURRENT-TARGET PREFERRED-TARGET openshift-storage local-pv-1a2b3c-worker-0-block local-pv-1a2b3c /dev/disk/by-id/scsi-0NVME_MODEL_abcde /dev/disk/by-id/scsi-2ace42e0035eabcde openshift-storage local-pv-4d5e6f-worker-1-block local-pv-4d5e6f /dev/disk/by-id/scsi-0NVME_MODEL_fghij /dev/disk/by-id/scsi-2ace42e0035efghij openshift-storage local-pv-7g8h9i-worker-2-block local-pv-7g8h9i MISSING /dev/disk/by-id/scsi-35000c500a1b2c3d4 ``` 3. Open the Custom Resource Definition, `localVolumeDeviceLink`: 1. Click **Ecosystem** > **Installed Operators**. 2. On the **Installed Operators** page, in the **Search by name** box, type "LocalVolumeDeviceLink". 3. Click **LocalVolumeDeviceLink**. Local Storage Operator creates `LocalVolumeDeviceLink` objects for each individual device (or partition or volume) it is managing. A cluster administrator has the option of specifying how LSO should handle symlinks for that particular device when underlying symlinks change because of unforeseen circumstances. 4. On the **Operator details** page, click the **YAML** tab. 5. Go to the `localVolumeDeviceLink.status` field and view its nested fields that are shown in the following table for a list of valid symlink targets, current link (`by-id`), and the generated preferred symlink. **`localVolumeDeviceLink.status` nested fields**
Status fields
Description
validLinkTargets
The full list of valid symlink targets, since there might be multiple by-id symlinks pointing to the same physical device.
currentLinkTarget
The by-id symlink currently used by the PV.
preferredLinkTarget
The preferred symlink LSO has determined that is less likely to change because of udev rules updates, firmware updates, and so on.
filesystemUUID
The corresponding UUID of the filesystem, if one is found.
```` Alternatively, you can view the status from the command line by running the following command: ```terminal $ oc get localvolumedevicelink local-pv-1a2b3c-worker-0-block -n openshift-storage -o jsonpath='{.status}' | jq ``` ```json title="Example" { "currentLinkTarget": "/dev/disk/by-id/scsi-0NVME_MODEL_abcde", "preferredLinkTarget": "/dev/disk/by-id/scsi-2ace42e0035eabcde", "validLinkTargets": [ "/dev/disk/by-id/nvme-eui.ace42e0035eabcde", "/dev/disk/by-id/scsi-0NVME_MODEL_abcde", "/dev/disk/by-id/scsi-2ace42e0035eabcde" ], "filesystemUUID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" } ``` 1. Got to the `localVolumeDeviceLink.spec.policy` field and set its value to one of the following options: * `CurrentLinkTarget`: Silences alerts and tells LSO to use the existing path. * `PreferredLinkTarget`: LSO recreates the symlink to point to the new, updated device path. Note that selecting this option does not mean that all symlinks are fixed for LV objects. Alternatively, you can set the policy from the command line: * To set the policy to `CurrentLinkTarget`, run the following command: ```terminal $ oc patch localvolumedevicelink local-pv-1a2b3c-worker-0-block -n openshift-storage \ --type merge -p '{"spec":{"policy":"CurrentLinkTarget"}}' ``` * To set the policy to `PreferredLinkTarget`, run the following command: ```terminal $ oc patch localvolumedevicelink local-pv-1a2b3c-worker-0-block -n openshift-storage \ --type merge -p '{"spec":{"policy":"PreferredLinkTarget"}}' ``` The following example output applies to both preceding commands: ```terminal title="Example" localvolumedevicelink.local.storage.openshift.io/local-pv-1a2b3c-worker-0-block patched ``` ```` 1. Click **Save**. 2. (Optional) Set `localVolumeDeviceLink.spec.policy` back to null. The default value of `localVolumeDeviceLink.spec.policy` is none, which means LSO is monitoring the volumes used by the OpenShift cluster, but does not automatically change symlinks for PVs if existing symlinks are broken or LSO finds a better symlink to use. In this state, LSO just alerts you if any anomaly is detected. 3. Click **Save**. ## Using tolerations with Local Storage Operator pods {#local-tolerations_persistent-storage-local} Configure tolerations in the `LocalVolume` resource to allow Local Storage Operator (LSO) pods to run on tainted nodes, enabling local storage provisioning on nodes that would otherwise repel all pods. Taints can be applied to nodes to prevent them from running general workloads. To allow the LSO to use tainted nodes, you must add tolerations to the `Pod` or `DaemonSet` definition. This allows the created resources to run on these tainted nodes. You apply tolerations to the LSO pod through the `LocalVolume` resource and apply taints to a node through the node specification. A taint on a node instructs the node to repel all pods that do not tolerate the taint. Using a specific taint that is not on other pods ensures that the LSO pod can also run on that node. > [!IMPORTANT] > Taints and tolerations consist of a key, value, and effect. As an argument, it is expressed as `key=value:effect`. An operator allows you to leave one of these parameters empty. **Prerequisites** - The LSO is installed. - Local disks are attached to OpenShift Container Platform nodes with a taint. - Tainted nodes are expected to provision local storage. **Procedure** 1. Modify the YAML file that defines the `Pod` and add the `LocalVolume` spec, as shown in the following example: ```yaml apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "local-disks" namespace: "openshift-local-storage" spec: tolerations: - key: localstorage operator: Equal value: "localstorage" storageClassDevices: - storageClassName: "local-sc" volumeMode: Block devicePaths: - /dev/xvdg ``` - `spec.tolerations.key`: Specifies the key that you added to the node. - `spec.tolerations.operator`: Specifies the `Equal` operator to require the `key`/`value` parameters to match. If operator is `Exists`, the system checks that the key exists and ignores the value. If operator is `Equal`, then the key and value must match. - `spec.tolerations.value`: Specifies the value `local` of the tainted node. - `spec.storageClassDevices.volumeMode`: Specifies the volume mode, either `Filesystem` or `Block`, defining the type of the local volumes. - `spec.storageClassDevices.devicePaths`: Specifies the path containing a list of local storage devices to choose from. 2. Optional: To create local persistent volumes on only tainted nodes, modify the YAML file and add the `LocalVolume` spec, as shown in the following example: ```yaml spec: tolerations: - key: node-role.kubernetes.io/master operator: Exists ``` **Results** The defined tolerations is passed to the resulting daemon sets, allowing the diskmaker and provisioner pods to be created for nodes that contain the specified taints. ## Local Storage Operator Metrics {#local-storage-metrics_persistent-storage-local} To understand storage utilization and troubleshoot provisioning issues, you can track Local Storage Operator (LSO) operations with available metrics that monitor disk discovery, persistent volume creation, unmatched disks, and orphaned symlinks. OpenShift Container Platform provides the following metrics for the LSO: - `lso_discovery_disk_count`: total number of discovered devices on each node - `lso_lvset_provisioned_PV_count`: total number of PVs created by `LocalVolumeSet` objects - `lso_lvset_unmatched_disk_count`: total number of disks that LSO did not select for provisioning because of mismatching criteria - `lso_lvset_orphaned_symlink_count`: number of devices with PVs that no longer match `LocalVolumeSet` object criteria - `lso_lv_orphaned_symlink_count`: number of devices with PVs that no longer match `LocalVolume` object criteria - `lso_lv_provisioned_PV_count`: total number of provisioned PVs for `LocalVolume` To use metrics, you must enable them first. For more information, see "Enabling Local Storage Operator Metrics". For more information about metrics, see "Accessing metrics as an administrator". ### Enabling Local Storage Operator Metrics {#local-storage-metrics-procedure_persistent-storage-local} To monitor Local Storage Operator (LSO) disk discovery, volume provisioning, and storage utilization, enable LSO metrics by configuring cluster monitoring on the Operator namespace. **Procedure** 1. Enable local metrics by doing one of the following: - When installing the LSO from the software catalog in the web console, select the **Enable Operator recommended cluster monitoring on this Namespace** checkbox. - Manually add the `openshift.io/cluster-monitoring=true` label to the Operator namespace by running the following command: ```terminal $ oc label ns/openshift-local-storage openshift.io/cluster-monitoring=true ``` ## Deleting the Local Storage Operator resources {#local-storage-deleting-resources-overview_persistent-storage-local} Occasionally you need to delete local volumes (LVs) and local volume sets (LVSs) when no longer needed or uninstall the Local Storage Operator, ensuring persistent volumes are properly released and backed up before removal to prevent data loss. ### Removing a local volume or local volume set {#local-removing-device_persistent-storage-local} Remove local volumes (LVs) and local volume sets (LVSs) when no longer needed by releasing bound persistent volumes (PV), deleting the LV or LVS, and cleaning up any PVs with `Retain` reclaim policy. **Prerequisites** - The PV must be in a `Released` or `Available` state. > [!WARNING] > Deleting a PV that is still in use can result in data loss or corruption. **Procedure** 1. If there are any bound PVs owned by the LV or LVS that is being deleted, delete the corresponding persistent volume claims (PVCs) to release the PVs: 1. To find bound PVs owned by a particular LV or LVS, run the following command: ```terminal $ oc get pv --selector storage.openshift.com/owner-name= ``` `` is the name of the LV or LVS. ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE local-pv-3fa1c73 5Gi RWO Delete Available slow 28s local-pv-1cec77cf 30Gi RWX Retain Bound openshift/storage my-sc 168d ``` Bound PVs have a status of `Bound` and their corresponding PVCs appear in the `CLAIM` column. In the preceding example, PV `local-pv-1cec77cf` is bound, and its PVC is `openshift/storage`. 2. Delete corresponding PVCs of bound PVs owned by the LV or LVS being deleted by running the following command: ```terminal $ oc delete pvc ``` In this example, you would delete PVC `openshift/storage`. 2. Delete the LVs or LVSs by running the applicable following command: - Command for deleting LV: ```terminal $ oc delete lv ``` or - Command for deleting LVS: ```terminal $ oc delete lvs ``` 3. If any PV owned by the LV or LVS has a `Retain` reclaim policy, back up any important data, and then delete the PV: > [!NOTE] > PVs with a `Delete` policy are automatically deleted when you delete the LVs or LVS. 1. To find PVs with `Retain` reclaim policy, run the following command: ```terminal $ oc get pv ``` ```terminal {title="Example output"} NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-1cec77cf 30Gi RWX Retain Available my-sc 168d ``` In this example, PV `local-pv-1cec77cf` has a `Retain` reclaim policy and needs to be manually deleted. 2. Back up any important data on this volume. 3. Delete the PV by running the following command: ```terminal $ oc delete pv ``` In this example, delete PV `local-pv-1cec77cf`. ### Uninstalling the Local Storage Operator {#local-storage-uninstall_persistent-storage-local} Safely uninstall the Local Storage Operator (LSO) when local storage persistent volumes (PVs) are no longer in use by removing all local volume resources, uninstalling the operator, deleting remaining PVs, and removing the project. > [!WARNING] > Uninstalling the LSO while local storage PVs are still in use is not recommended. Although the PVs remain after removing the Operator, there might be indeterminate behavior if the Operator is uninstalled and reinstalled without removing the PVs and local storage resources. **Prerequisites** - Access to the OpenShift Container Platform web console. **Procedure** 1. Delete any local volume resources installed in the project, such as `localvolume`, `localvolumeset`, and `localvolumediscovery` by running the following commands: ```terminal $ oc delete localvolume --all --all-namespaces ``` ```terminal $ oc delete localvolumeset --all --all-namespaces ``` ```terminal $ oc delete localvolumediscovery --all --all-namespaces ``` 2. Uninstall the LSO from the OpenShift Container Platform web console. 1. Log in to the OpenShift Container Platform web console. 2. Go to **Ecosystem** → **Installed Operators**. 3. Type **Local Storage** into the filter box to locate the LSO. 4. Click the **Options** menu ![](/openshift-docs-markdown/images/kebab.png "Options menu") at the end of the LSO. 5. Click **Uninstall Operator**. 6. Click **Remove** in the window that appears. 3. The PVs created by the LSO remain in the cluster until deleted. After these volumes are no longer in use, delete them by running the following command: ```terminal $ oc delete pv ``` 4. Delete the `openshift-local-storage` project by running the following command: ```terminal $ oc delete project openshift-local-storage ``` **Additional resources** {._additional-resources} - [Installing the Local Storage Operator](/openshift-docs-markdown/storage/persistent_storage_local/persistent-storage-local#local-storage-install-overview_persistent-storage-local) - [Enabling Local Storage Operaator Metric](/openshift-docs-markdown/storage/persistent_storage_local/persistent-storage-local#local-storage-metrics-procedure_persistent-storage-local) - [Accessing metrics as an administrator](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/accessing_metrics/accessing-metrics-as-an-administrator)