--- title: AWS Elastic File Service CSI Driver Operator --- # AWS Elastic File Service CSI Driver Operator {#persistent-storage-csi-aws-efs} You can provision and manage AWS Elastic File System (EFS) storage in OpenShift Container Platform by using the AWS EFS Container Storage Interface (CSI) Driver Operator and driver, which provide dynamic volume provisioning and eliminate the need to pre-provision storage. ## Overview {#efs-driver-overview_persistent-storage-csi-aws-efs} OpenShift Container Platform is capable of provisioning persistent volumes (PVs) using the Container Storage Interface (CSI) driver for AWS Elastic File Service (EFS). Familiarity with persistent storage and configuring CSI volumes is recommended when working with a CSI Operator and driver. For more information, see "Understanding persistent storage" and "Configuring CSI volumes". After installing the AWS EFS CSI Driver Operator, OpenShift Container Platform installs the AWS EFS CSI Operator and the AWS EFS CSI driver by default in the `openshift-cluster-csi-drivers` namespace. This allows the AWS EFS CSI Driver Operator to create CSI-provisioned PVs that mount to AWS EFS assets. - The *AWS EFS CSI Driver Operator*, after being installed, does not create a storage class by default to use to create persistent volume claims (PVCs). However, you can manually create the AWS EFS `StorageClass`. The AWS EFS CSI Driver Operator supports dynamic volume provisioning by allowing storage volumes to be created on-demand. This eliminates the need for cluster administrators to pre-provision storage. - The *AWS EFS CSI driver* enables you to create and mount AWS EFS PVs. **Additional resources** {._additional-resources} - [Understanding persistent storage](/openshift-docs-markdown/storage/understanding-persistent-storage#understanding-persistent-storage) - [Configuring CSI volumes](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi#persistent-storage-csi) ## About CSI {#csi-about_persistent-storage-csi-aws-efs} The Container Storage Interface (CSI) enables storage vendors to deliver plugins through a standard interface without modifying Kubernetes core code, replacing traditional embedded storage drivers. CSI Operators give OpenShift Container Platform users storage options, such as volume snapshots, that are not possible with in-tree volume plugins. ## Setting up the AWS EFS CSI Driver Operator {#persistent-storage-efs-csi-driver-operator-setup_persistent-storage-csi-aws-efs} To enable AWS Elastic File System (EFS) storage in your cluster, complete the setup process by obtaining necessary credentials, installing the operator, and installing the driver. **Procedure** 1. If you are using AWS EFS with AWS Secure Token Service (STS), obtain a role Amazon Resource Name (ARN) for STS. This is required for installing the AWS EFS CSI Driver Operator. 2. Install the AWS EFS CSI Driver Operator. 3. Install the AWS EFS CSI Driver. ### Obtaining a role Amazon Resource Name for Security Token Service {#efs-sts_persistent-storage-csi-aws-efs} To configure the AWS Elastic File System (EFS) Container Storage Interface (CSI) Driver Operator on clusters using Security Token Service (STS), obtain a role Amazon Resource Name (ARN) using the Cloud Credential Operator utility before installation. > [!IMPORTANT] > Perform this procedure before you install the AWS EFS CSI Driver Operator (see *Installing the AWS EFS CSI Driver Operator* procedure). You can obtain the ARN role in multiple ways. The following procedure shows one method that uses the same concept and CCO utility (`ccoctl`) binary tool as cluster installation. > [!NOTE] > If you are using One Zone file system, you need to create two `CredentialRequests`, one for the controller and one for the driver node. For more information, see Section *Setting up One Zone file systems with STS*. **Prerequisites** - Access to the cluster as a user with the cluster-admin role. - AWS account credentials **Procedure** 1. Extract the `ccoctl` from the OpenShift Container Platform release image, which you used to install the cluster with STS. For more information, see "Configuring the Cloud Credential Operator utility". 2. Create and save an EFS `CredentialsRequest` YAML file, such as shown in the following example, and then place it in the `credrequests` directory: ```yaml {title="Example"} apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: openshift-aws-efs-csi-driver namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - action: - elasticfilesystem:* effect: Allow resource: '*' secretRef: name: aws-efs-cloud-credentials namespace: openshift-cluster-csi-drivers serviceAccountNames: - aws-efs-csi-driver-operator - aws-efs-csi-driver-controller-sa ``` 3. Run the `ccoctl` tool to generate a new IAM role in AWS, and create a YAML file for it in the local file system (`/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml`). ```terminal $ ccoctl aws create-iam-roles --name= --region= --credentials-requests-dir=/credrequests --identity-provider-arn=arn:aws:iam:::oidc-provider/-oidc.s3..amazonaws.com ``` - `name=` is the name used to tag any cloud resources that are created for tracking. - `region=` is the AWS region where cloud resources are created. - `dir=/credrequests` is the directory containing the EFS CredentialsRequest file in previous step. - `` is the AWS account ID. ```terminal {title="Example"} $ ccoctl aws create-iam-roles --name my-aws-efs --credentials-requests-dir credrequests --identity-provider-arn arn:aws:iam::123456789012:oidc-provider/my-aws-efs-oidc.s3.us-east-2.amazonaws.com ``` ```terminal {title="Example output"} 2022/03/21 06:24:44 Role arn:aws:iam::123456789012:role/my-aws-efs -openshift-cluster-csi-drivers-aws-efs-cloud- created 2022/03/21 06:24:44 Saved credentials configuration to: /manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml 2022/03/21 06:24:45 Updated Role policy for Role my-aws-efs-openshift-cluster-csi-drivers-aws-efs-cloud- ``` 4. Copy the role ARN from the first line of the *Example output* in the preceding step. The role ARN is between "Role" and "created". In this example, the role ARN is "arn:aws:iam::123456789012:role/my-aws-efs -openshift-cluster-csi-drivers-aws-efs-cloud". You will need the role ARN when you install the AWS EFS CSI Driver Operator. **Next steps** Install the AWS EFS CSI Driver Operator. For information, see Inst"alling the AWS EFS CSI Driver Operator". **Additional resources** {._additional-resources} - [Installing the AWS EFS CSI Driver Operator](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi-aws-efs#persistent-storage-csi-olm-operator-install_persistent-storage-csi-aws-efs) - [Configuring the Cloud Credential Operator utility](/openshift-docs-markdown/installing/installing_aws/ipi/installing-aws-customizations#cco-ccoctl-configuring_installing-aws-customizations) - [Installing the AWS EFS CSI Driver](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi-aws-efs#persistent-storage-csi-efs-driver-install_persistent-storage-csi-aws-efs) ### Installing the AWS EFS CSI Driver Operator {#persistent-storage-csi-olm-operator-install_persistent-storage-csi-aws-efs} Install and configure the AWS EFS Container Storage Interface (CSI) Driver Operator to enable AWS EFS storage in your cluster. This Red Hat Operator is not installed by default and requires manual installation. **Prerequisites** - Access to the OpenShift Container Platform web console. **Procedure** 1. Log in to the web console. 2. Install the AWS EFS CSI Operator: 1. Click **Ecosystem** → **Software Catalog**. 2. Locate the AWS EFS CSI Operator by typing **AWS EFS CSI** in the filter box. 3. Click the **AWS EFS CSI Driver Operator** button. > [!IMPORTANT] > Be sure to select the **AWS EFS CSI Driver Operator** and not the **AWS EFS Operator**. The **AWS EFS Operator** is a community Operator and is not supported by Red Hat. 1. On the **AWS EFS CSI Driver Operator** page, click **Install**. 2. On the **Install Operator** page, ensure that: - If you are using AWS EFS with AWS Secure Token Service (STS), in the **role ARN** field, enter the ARN role copied from the last step of the *Obtaining a role Amazon Resource Name for Security Token Service* procedure. - **All namespaces on the cluster (default)** is selected. - **Installed Namespace** is set to **openshift-cluster-csi-drivers**. 3. Click **Install**. After the installation finishes, the AWS EFS CSI Operator is listed in the **Installed Operators** section of the web console. **Next steps** Install the AWS EFS CSI Driver. ### Installing the AWS EFS CSI Driver {#persistent-storage-csi-efs-driver-install_persistent-storage-csi-aws-efs} After installing the Container Storage Interface (CSI) Driver Operator (a Red Hat operator), you need to install the AWS EFS CSI driver. **Prerequisites** - Access to the OpenShift Container Platform web console. **Procedure** 1. Click **Administration** → **CustomResourceDefinitions** → **ClusterCSIDriver**. 2. On the **Instances** tab, click **Create ClusterCSIDriver**. 3. Use the following YAML file: ```yaml apiVersion: operator.openshift.io/v1 kind: ClusterCSIDriver metadata: name: efs.csi.aws.com spec: managementState: Managed ``` Where `metadata.name` is the storage provisioner name. 4. Click **Create**. 5. Wait for the following Conditions to change to a "True" status: - AWSEFSDriverNodeServiceControllerAvailable - AWSEFSDriverControllerServiceControllerAvailable ## About the AWS EFS storage class {#storage-create-storage-class_persistent-storage-csi-aws-efs} To enable dynamic provisioning of persistent volumes, create a storage class that defines storage characteristics and allows users to automatically provision volumes on-demand. The AWS Elastic File System (EFS) Container Storage Interface (CSI) Driver Operator (a Red Hat operator), after being installed, does not create a storage class by default. However, you can manually create the AWS EFS storage class. ### Creating the AWS EFS storage class using the console {#storage-create-storage-class-console_persistent-storage-csi-aws-efs} To enable dynamic provisioning of AWS Elastic File System (EFS) volumes using the console, create a `StorageClass` object that defines file system parameters, permissions, and access point configuration. **Procedure** 1. In the OpenShift Container Platform web console, click **Storage** → **StorageClasses**. 2. On the **StorageClasses** page, click **Create StorageClass**. 3. On the **StorageClass** page, perform the following steps: 1. Enter a name to reference the storage class. 2. Optional: Enter the description. 3. Select the reclaim policy. 4. Select **`efs.csi.aws.com`** from the **Provisioner** drop-down list. 5. Optional: Set the configuration parameters for the selected provisioner. 4. Click **Create**. ### Creating the AWS EFS storage class using the CLI {#storage-create-storage-class-cli_persistent-storage-csi-aws-efs} To enable dynamic provisioning of AWS Elastic File System (EFS) volumes by using the command line, create a `StorageClass` object that defines file system parameters, permissions, and access point configuration. **Procedure** - Create a `StorageClass` object using the following example YAML file: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com parameters: provisioningMode: efs-ap fileSystemId: fs-a5324911 directoryPerms: "700" gidRangeStart: "1000" gidRangeEnd: "2000" basePath: "/dynamic_provisioning" ``` - `parameters.provisioningMode`: Must be set to `efs-ap` to enable dynamic provisioning. - `parameters.fileSystemId`: Must be the ID of the EFS volume created manually. - `parameters.directoryPerms`: Is the default permission of the root directory of the volume. In this example, the volume is accessible only by the owner. - `parameters.gidRangeStart` and `parameters.gidRangeEnd`: Set the range of POSIX Group IDs (GIDs) that are used to set the GID of the AWS access point. If not specified, the default range is 50000-7000000. Each provisioned volume, and thus AWS access point, is assigned a unique GID from this range. - `parameters.basePath`: Is the directory on the EFS volume that is used to create dynamically provisioned volumes. In this case, a PV is provisioned as “/dynamic_provisioning/” on the EFS volume. Only the subdirectory is mounted to pods that use the PV. > [!NOTE] > A cluster admin can create several `StorageClass` objects, each using a different EFS volume. **Additional resources** {._additional-resources} - [AWS EFS CSI Driver Operator](https://github.com/openshift/aws-efs-csi-driver-operator) ## AWS EFS CSI cross account support {#persistent-storage-csi-efs-cross-account_persistent-storage-csi-aws-efs} To mount AWS Elastic File System (EFS) volumes across AWS accounts, configure cross-account support that allows your OpenShift Container Platform cluster in one account to access file systems in another account. **Prerequisites** - Access to an OpenShift Container Platform cluster with administrator rights - Two valid AWS accounts - The EFS CSI Operator has been installed. For information about installing the EFS CSI Operator, see the *Installing the AWS EFS CSI Driver Operator* section. - Both the OpenShift Container Platform cluster and EFS file system must be located in the same AWS region. - Ensure that the two virtual private clouds (VPCs) used in the following procedure use different network Classless Inter-Domain Routing (CIDR) ranges. - Access to OpenShift Container Platform CLI (`oc`). - Access to AWS CLI. - Access to `jq` command-line JSON processor. The following procedure explains how to set up: - OpenShift Container Platform AWS Account A: Contains a Red Hat OpenShift Container Platform cluster v4.16, or later, deployed within a VPC - AWS Account B: Contains a VPC (including subnets, route tables, and network connectivity). The EFS filesystem will be created in this VPC. **Procedure** 1. Set up the environment: 1. Configure environment variables by running the following commands: ```terminal export CLUSTER_NAME="" export AWS_REGION="" export AWS_ACCOUNT_A_ID="" export AWS_ACCOUNT_B_ID="" export AWS_ACCOUNT_A_VPC_CIDR="" export AWS_ACCOUNT_B_VPC_CIDR="" export AWS_ACCOUNT_A_VPC_ID="" export AWS_ACCOUNT_B_VPC_ID="" export SCRATCH_DIR="" export CSI_DRIVER_NAMESPACE="openshift-cluster-csi-drivers" export AWS_PAGER="" ``` - ``: Cluster name of choice. - ``: AWS region of choice. - ``: AWS Account A ID. - ``: AWS Account B ID. - ``: CIDR range of VPC in Account A. - ``: CIDR range of VPC in Account B. - ``: VPC ID in Account A (cluster) - ``: VPC ID in Account B (EFS cross account) - ``: Any writeable directory of choice to use to store temporary files. - `CSI_DRIVER_NAMESPACE`: If your driver is installed in a non-default namespace, change this value. - `AWS_PAGER`: Makes AWS CLI output everything directly to stdout. 2. Create the working directory by running the following command: ```terminal mkdir -p $SCRATCH_DIR ``` 3. Verify cluster connectivity by running the following command in the OpenShift Container Platform CLI: ```terminal $ oc whoami ``` 4. Determine the OpenShift Container Platform cluster type and set node selector: The EFS cross account feature requires assigning AWS IAM policies to nodes running EFS CSI controller pods. However, this is not consistent for every OpenShift Container Platform type. - If your cluster is deployed as a Hosted Control Plane (HyperShift), set the `NODE_SELECTOR` environment variable to hold the worker node label by running the following command: ```terminal export NODE_SELECTOR=node-role.kubernetes.io/worker ``` - For all other OpenShift Container Platform types, set the `NODE_SELECTOR` environment variable to hold the master node label by running the following command: ```terminal export NODE_SELECTOR=node-role.kubernetes.io/master ``` 5. Configure AWS CLI profiles as environment variables for account switching by running the following commands: ```terminal export AWS_ACCOUNT_A="" export AWS_ACCOUNT_B="" ``` 6. Ensure that your AWS CLI is configured with JSON output format as the default for both accounts by running the following commands: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A} aws configure get output export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B} aws configure get output ``` If the preceding commands return: - **No value**: The default output format is already set to JSON and no changes are required. - **Any value**: Reconfigure your AWS CLI to use JSON format. For information about changing output formats, see *Setting the output format in the AWS CLI* in the AWS documentation. 7. Unset `AWS_PROFILE` in your shell to prevent conflicts with `AWS_DEFAULT_PROFILE` by running the following command: ```terminal unset AWS_PROFILE ``` 2. Configure the AWS Account B IAM roles and policies: 1. Switch to your Account B profile by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B} ``` 2. Define the IAM role name for the EFS CSI Driver Operator by running the following command: ```terminal export ACCOUNT_B_ROLE_NAME=${CLUSTER_NAME}-cross-account-aws-efs-csi-operator ``` 3. Create the IAM trust policy file by running the following command: ```terminal cat < $SCRATCH_DIR/AssumeRolePolicyInAccountB.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::${AWS_ACCOUNT_A_ID}:root" }, "Action": "sts:AssumeRole", "Condition": {} } ] } EOF ``` 4. Create the IAM role for the EFS CSI Driver Operator by running the following command: ```terminal ACCOUNT_B_ROLE_ARN=$(aws iam create-role \ --role-name "${ACCOUNT_B_ROLE_NAME}" \ --assume-role-policy-document file://$SCRATCH_DIR/AssumeRolePolicyInAccountB.json \ --query "Role.Arn" --output text) \ && echo $ACCOUNT_B_ROLE_ARN ``` 5. Create the IAM policy file by running the following command: ```terminal cat << EOF > $SCRATCH_DIR/EfsPolicyInAccountB.json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "ec2:DescribeNetworkInterfaces", "ec2:DescribeSubnets" ], "Resource": "*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "elasticfilesystem:DescribeMountTargets", "elasticfilesystem:DeleteAccessPoint", "elasticfilesystem:ClientMount", "elasticfilesystem:DescribeAccessPoints", "elasticfilesystem:ClientWrite", "elasticfilesystem:ClientRootAccess", "elasticfilesystem:DescribeFileSystems", "elasticfilesystem:CreateAccessPoint", "elasticfilesystem:TagResource" ], "Resource": "*" } ] } EOF ``` 6. Create the IAM policy by running the following command: ```terminal ACCOUNT_B_POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-efs-csi-policy" \ --policy-document file://$SCRATCH_DIR/EfsPolicyInAccountB.json \ --query 'Policy.Arn' --output text) \ && echo ${ACCOUNT_B_POLICY_ARN} ``` 7. Attach the policy to the role by running the following command: ```terminal aws iam attach-role-policy \ --role-name "${ACCOUNT_B_ROLE_NAME}" \ --policy-arn "${ACCOUNT_B_POLICY_ARN}" ``` 3. Configure the AWS Account A IAM roles and policies: 1. Switch to your Account A profile by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A} ``` 2. Create the IAM policy document by running the following command: ```terminal cat << EOF > $SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "${ACCOUNT_B_ROLE_ARN}" } ] } EOF ``` 3. In AWS Account A, attach the AWS-managed policy "AmazonElasticFileSystemClientFullAccess" to the OpenShift Container Platform cluster master role by running the following command: ```terminal EFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN=arn:aws:iam::aws:policy/AmazonElasticFileSystemClientFullAccess declare -A ROLE_SEEN for NODE in $(oc get nodes --selector="${NODE_SELECTOR}" -o jsonpath='{.items[*].metadata.name}'); do INSTANCE_PROFILE=$(aws ec2 describe-instances \ --filters "Name=private-dns-name,Values=${NODE}" \ --query 'Reservations[].Instances[].IamInstanceProfile.Arn' \ --output text | awk -F'/' '{print $NF}' | xargs) MASTER_ROLE_ARN=$(aws iam get-instance-profile \ --instance-profile-name "${INSTANCE_PROFILE}" \ --query 'InstanceProfile.Roles[0].Arn' \ --output text | xargs) MASTER_ROLE_NAME=$(echo "${MASTER_ROLE_ARN}" | awk -F'/' '{print $NF}' | xargs) echo "Checking role: '${MASTER_ROLE_NAME}'" if [[ -n "${ROLE_SEEN[$MASTER_ROLE_NAME]:-}" ]]; then echo "Already processed role: '${MASTER_ROLE_NAME}', skipping." continue fi ROLE_SEEN["$MASTER_ROLE_NAME"]=1 echo "Assigning policy ${EFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN} to role ${MASTER_ROLE_NAME}" aws iam attach-role-policy --role-name "${MASTER_ROLE_NAME}" --policy-arn "${EEFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN}" done ``` 4. Attach the policy to the IAM entity to allow role assumption: This step depends on your cluster configuration. In both of the following scenarios, the EFS CSI Driver Operator uses an entity to authenticate to AWS, and this entity must be granted permission to assume roles in Account B. If your cluster: - **Does not have STS enabled**: The EFS CSI Driver Operator uses an IAM User entity for AWS authentication. Continue with the step "Attach policy to IAM User to allow role assumption". - **Has STS enabled**: The EFS CSI Driver Operator uses an IAM role entity for AWS authentication. Continue with the step "Attach policy to IAM Role to allow role assumption". 5. Attach policy to IAM User to allow role assumption 1. Identify the IAM User used by the EFS CSI Driver Operator by running the following command: ```terminal EFS_CSI_DRIVER_OPERATOR_USER=$(oc -n openshift-cloud-credential-operator get credentialsrequest/openshift-aws-efs-csi-driver -o json | jq -r '.status.providerStatus.user') ``` 2. Attach the policy to the IAM user by running the following command: ```terminal aws iam put-user-policy \ --user-name "${EFS_CSI_DRIVER_OPERATOR_USER}" \ --policy-name efs-cross-account-inline-policy \ --policy-document file://$SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json ``` 6. Attach the policy to the IAM role to allow role assumption: 1. Identify the IAM role name currently used by the EFS CSI Driver Operator by running the following command: ```terminal EFS_CSI_DRIVER_OPERATOR_ROLE=$(oc -n ${CSI_DRIVER_NAMESPACE} get secret/aws-efs-cloud-credentials -o jsonpath='{.data.credentials}' | base64 -d | grep role_arn | cut -d'/' -f2) && echo ${EFS_CSI_DRIVER_OPERATOR_ROLE} ``` 2. Attach the policy to the IAM role used by the EFS CSI Driver Operator by running the following command: ```terminal aws iam put-role-policy \ --role-name "${EFS_CSI_DRIVER_OPERATOR_ROLE}" \ --policy-name efs-cross-account-inline-policy \ --policy-document file://$SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json ``` 7. Configure VPC peering: 1. Initiate a peering request from Account A to Account B by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A} PEER_REQUEST_ID=$(aws ec2 create-vpc-peering-connection --vpc-id "${AWS_ACCOUNT_A_VPC_ID}" --peer-vpc-id "${AWS_ACCOUNT_B_VPC_ID}" --peer-owner-id "${AWS_ACCOUNT_B_ID}" --query VpcPeeringConnection.VpcPeeringConnectionId --output text) ``` 2. Accept the peering request from Account B by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B} aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id "${PEER_REQUEST_ID}" ``` 3. Retrieve the route table IDs for Account A and add routes to the Account B VPC by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A} for NODE in $(oc get nodes --selector=node-role.kubernetes.io/worker | tail -n +2 | awk '{print $1}') do SUBNET=$(aws ec2 describe-instances --filters "Name=private-dns-name,Values=$NODE" --query 'Reservations[*].Instances[*].NetworkInterfaces[*].SubnetId' | jq -r '.[0][0][0]') echo SUBNET is ${SUBNET} ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=${SUBNET}" --query 'RouteTables[*].RouteTableId' | jq -r '.[0]') echo Route table ID is $ROUTE_TABLE_ID aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_B_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID} done ``` 4. Retrieve the route table IDs for Account B and add routes to the Account A VPC by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B} for ROUTE_TABLE_ID in $(aws ec2 describe-route-tables --filters "Name=vpc-id,Values=${AWS_ACCOUNT_B_VPC_ID}" --query "RouteTables[].RouteTableId" | jq -r '.[]') do echo Route table ID is $ROUTE_TABLE_ID aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_A_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID} done ``` 8. Configure security groups in Account B to allow NFS traffic from Account A to EFS: 1. Switch to your Account B profile by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B} ``` 2. Configure the VPC security groups for EFS access by running the following command: ```terminal SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values="${AWS_ACCOUNT_B_VPC_ID}" | jq -r '.SecurityGroups[].GroupId') aws ec2 authorize-security-group-ingress \ --group-id "${SECURITY_GROUP_ID}" \ --protocol tcp \ --port 2049 \ --cidr "${AWS_ACCOUNT_A_VPC_CIDR}" | jq . ``` 9. Create a region-wide EFS filesystem in Account B: 1. Switch to your Account B profile by running the following command: ```terminal export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B} ``` 2. Create a region-wide EFS file system by running the following command: ```terminal CROSS_ACCOUNT_FS_ID=$(aws efs create-file-system --creation-token efs-token-1 \ --region ${AWS_REGION} \ --encrypted | jq -r '.FileSystemId') \ && echo $CROSS_ACCOUNT_FS_ID ``` 3. Configure region-wide mount targets for EFS by running the following command: ```terminal for SUBNET in $(aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${AWS_ACCOUNT_B_VPC_ID}" \ --region ${AWS_REGION} \ | jq -r '.Subnets.[].SubnetId'); do \ MOUNT_TARGET=$(aws efs create-mount-target --file-system-id ${CROSS_ACCOUNT_FS_ID} \ --subnet-id ${SUBNET} \ --region ${AWS_REGION} \ | jq -r '.MountTargetId'); \ echo ${MOUNT_TARGET}; \ done ``` This creates a mount point in each subnet of your VPC. 10. Configure the EFS Operator for cross-account access: 1. Define custom names for the secret and storage class that you will create in subsequent steps by running the following command: ```terminal export SECRET_NAME=my-efs-cross-account export STORAGE_CLASS_NAME=efs-sc-cross ``` 2. Create a secret that references the role ARN in Account B by running the following command in the OpenShift Container Platform CLI: ```terminal oc create secret generic ${SECRET_NAME} -n ${CSI_DRIVER_NAMESPACE} --from-literal=awsRoleArn="${ACCOUNT_B_ROLE_ARN}" ``` 3. Grant the CSI driver controller access to the newly created secret by running the following commands in the OpenShift Container Platform CLI: ```terminal oc -n ${CSI_DRIVER_NAMESPACE} create role access-secrets --verb=get,list,watch --resource=secrets oc -n ${CSI_DRIVER_NAMESPACE} create rolebinding --role=access-secrets default-to-secrets --serviceaccount=${CSI_DRIVER_NAMESPACE}:aws-efs-csi-driver-controller-sa ``` 4. Create a new storage class that references the EFS ID from Account B and the secret created previously by running the following command in the OpenShift Container Platform CLI: ```terminal cat << EOF | oc apply -f - kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: ${STORAGE_CLASS_NAME} provisioner: efs.csi.aws.com parameters: provisioningMode: efs-ap fileSystemId: ${CROSS_ACCOUNT_FS_ID} directoryPerms: "700" gidRangeStart: "1000" gidRangeEnd: "2000" basePath: "/dynamic_provisioning" csi.storage.k8s.io/provisioner-secret-name: ${SECRET_NAME} csi.storage.k8s.io/provisioner-secret-namespace: ${CSI_DRIVER_NAMESPACE} EOF ``` **Additional resources** {._additional-resources} - [Setting the output format in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output-format.html) ## One Zone file systems {#efs-one-zone-intro_persistent-storage-csi-aws-efs} You can use AWS Elastic File System (EFS) One Zone file systems to store data redundantly within a single Availability Zone (AZ), offering a lower-cost storage option compared to multi-AZ regional storage. ### One Zone file systems overview {#efs-one-zone-overview_persistent-storage-csi-aws-efs} AWS Elastic File System (EFS) One Zone contrasts with the default EFS storage option, which stores data redundantly across multiple AZs within a region. Clusters upgraded from OpenShift Container Platform 4.19 are compatible with the regional EFS volumes. > [!NOTE] > Dynamic provisioning of One Zone volumes is supported only in single-zone clusters. All nodes in the cluster must be in the same AZ as the EFS volume that is used for the dynamic provisioning. > > Manually provisioned One Zone volumes in regional clusters is supported, assuming that the persistent volumes (PVs) have correct `spec.nodeAffinity` that indicates the zone that the volume is in. For Cloud Credential Operator (CCO) Mint mode or Passthrough, no extra configuration is required. However, for Security Token Service (STS), use the procedure in Section *Setting up One Zone file systems with STS*. ### Setting up One Zone file systems with STS {#efs-one-zone-procedure_persistent-storage-csi-aws-efs} Configure separate credential requests and role ARNs for the controller and driver nodes to use AWS Elastic File System (EFS) One Zone file systems with Security Token Service (STS) authentication. The following procedure explains how to set up AWS One Zone file systems with Security Token Service (STS). **Prerequisites** - Access to the cluster as a user with the cluster-admin role. - AWS account credentials **Procedure** 1. Create **two** `CredentialsRequests` in the `credrequests` directory following the procedure under Section *Obtaining a role Amazon Resource Name for Security Token Service*: - For the **controller** `CredentialsRequest`, follow the procedure without any changes. - For the **driver node** `CredentialsRequest` use the following example file: ```yaml {title="Example CredentialsRequest YAML file for driver node"} apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: annotations: credentials.openshift.io/role-arns-vars: NODE_ROLEARN name: openshift-aws-efs-csi-driver-node namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - action: - elasticfilesystem:DescribeMountTargets - ec2:DescribeAvailabilityZones effect: Allow resource: '*' secretRef: name: node-aws-efs-cloud-credentials namespace: openshift-cluster-csi-drivers serviceAccountNames: - aws-efs-csi-driver-node-sa ``` Set `metadata.annotations.credentials.openshift.io/role-arns-vars` to `NODE_ROLEARN`. ```terminal {title="Example ccoctl output"} 2025/08/26 14:05:24 Role arn:aws:iam::269733383066:role/my-arn-1-blll6-openshift-cluster-csi-drivers-aws-efs-cloud-cre created 2025/08/26 14:05:24 Saved credentials configuration to: /home/my-arn/project/go/src/github.com/openshift/myinst/aws-sts-compact-1/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml 2025/08/26 14:05:24 Updated Role policy for Role my-arn-1-blll6-openshift-cluster-csi-drivers-aws-efs-cloud-cre 2025/08/26 14:05:24 Role arn:aws:iam::269733383066:role/my-arn-1-blll6-openshift-cluster-csi-drivers-node-aws-efs-clou created 2025/08/26 14:05:24 Saved credentials configuration to: manifests/openshift-cluster-csi-drivers-node-aws-efs-cloud-credentials-credentials.yaml 2025/08/26 14:05:24 Updated Role policy for Role my-arn-1-blll6-openshift-cluster-csi-drivers-node-aws-efs-clou ``` In this example: - The first line shows the Controller Amazon Resource Name (ARN). - The fifth line shows the Driver node ARN. 2. Install the AWS EFS CSI driver using the controller ARN created earlier in this procedure. 3. Edit the operator’s subscription and add `NODE_ROLEARN` with the driver node’s ARN by running a command similar to the following: ```terminal $ oc -n openshift-cluster-csi-drivers edit subscription aws-efs-csi-driver-operator ... config: env: - name: ROLEARN value: arn:aws:iam::269733383066:role/my-arn-1-blll6-openshift-cluster-csi-drivers-aws-efs-cloud-cre - name: NODE_ROLEARN value: arn:aws:iam::269733383066:role/my-arn-1-blll6-openshift-cluster-csi-drivers-node-aws-efs-clou ... ``` - `ROLEARN` `value` is the Controller ARN, which already exists. - `NODE_ROLEARN` `value` is the Driver node ARN. ## Dynamic provisioning for Amazon Elastic File Storage {#csi-dynamic-provisioning-aws-efs_persistent-storage-csi-aws-efs} To dynamically provision persistent volumes (PVs) as subdirectories of an existing Elastic File System (EFS) volume, create persistent volume claims (PVCs) referencing your EFS storage class to enable multiple independent volumes sharing the same resource. The AWS EFS Container Storage Interface (CSI) driver supports a different form of dynamic provisioning than other CSI drivers. It provisions new PVs as subdirectories of a pre-existing EFS volume. The PVs are independent of each other. However, they all share the same EFS volume. When the volume is deleted, all PVs provisioned out of it are deleted too. The EFS CSI driver creates an AWS Access Point for each such subdirectory. Due to AWS AccessPoint limits, you can only dynamically provision 1000 PVs from a single `StorageClass`/EFS volume. > [!IMPORTANT] > Note that `PVC.spec.resources` is not enforced by EFS. > > In the example below, you request 5 GiB of space. However, the created PV is limitless and can store any amount of data (like petabytes). A broken application, or even a rogue application, can cause significant expenses when it stores too much data on the volume. > > Using monitoring of EFS volume sizes in AWS is strongly recommended. If you have problems setting up dynamic provisioning, see *AWS EFS troubleshooting*. **Prerequisites** - You have created Amazon Elastic File Storage (Amazon EFS) volumes. - You have created the AWS EFS storage class. **Procedure** - Create a PVC (or StatefulSet or Template) as usual, referring to the `StorageClass` created previously. ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test spec: storageClassName: efs-sc accessModes: - ReadWriteMany resources: requests: storage: 5Gi ``` **Additional resources** {._additional-resources} - [AWS EFS CSI driver](https://github.com/openshift/aws-efs-csi-driver) - [Creating the AWS EFS storage class](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi-aws-efs#storage-create-storage-class_persistent-storage-csi-aws-efs) ## Creating static PVs with Amazon Elastic File Storage {#efs-create-static-pv_persistent-storage-csi-aws-efs} To mount an entire Amazon Elastic File Storage (EFS) volume as a single persistent volume without dynamic provisioning, create a static persistent volume (PV) that allows pods to access the full volume. If you have problems setting up static PVs, see "AWS EFS troubleshooting". **Prerequisites** - You have created Amazon EFS volumes. **Procedure** - Create the PV using the following YAML file: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: efs-pv spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteMany - ReadWriteOnce persistentVolumeReclaimPolicy: Retain csi: driver: efs.csi.aws.com volumeHandle: fs-ae66151a volumeAttributes: encryptInTransit: "false" ``` - `spec.capacity` does not have any meaning and is ignored by the CSI driver. It is used only when binding to a PVC. Applications can store any amount of data to the volume. - `spec.csi.volumeHandle` must be the same ID as the EFS volume you created in AWS. If you are providing your own access point, `volumeHandle` should be `::`. For example: `fs-6e633ada::fsap-081a1d293f0004630`. - `spec.csi.volumeAttributes.encryptInTransit`: If desired, you can disable encryption in transit. Encryption is enabled by default. ## Amazon Elastic File Storage security {#efs-security_persistent-storage-csi-aws-efs} When using Amazon Elastic File Storage (EFS) with access points, understand that file permissions are controlled by the access point rather than Kubernetes FSGroup settings, allowing any pod with access to read all files. When using access points, for example, by using dynamic provisioning as described earlier, Amazon automatically replaces GIDs on files with the GID of the access point. In addition, EFS considers the user ID, group ID, and secondary group IDs of the access point when evaluating file system permissions. EFS ignores the NFS client’s IDs. For more information about access points, see "Working with access points". As a consequence, EFS volumes silently ignore FSGroup; OpenShift Container Platform is not able to replace the GIDs of files on the volume with FSGroup. Any pod that can access a mounted EFS access point can access any file on it. Unrelated to this, encryption in transit is enabled by default. For more information, see "Encrypting data in transit". **Additional resources** {._additional-resources} - [Working with access points](https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html) - [Encrypting data in transit](https://docs.aws.amazon.com/efs/latest/ug/encryption-in-transit.html) ## AWS EFS storage CSI usage metrics {#efs-metrics-overview_persistent-storage-csi-aws-efs} Amazon Web Services (AWS) Elastic File System (EFS) storage Container Storage Interface (CSI) usage metrics allow you to monitor how much space is used by either dynamically or statically provisioned EFS volumes. > [!IMPORTANT] > This features is disabled by default, because turning on metrics can lead to performance degradation. The AWS EFS usage metrics feature collects volume metrics in the AWS EFS CSI Driver by recursively walking through the files in the volume. Because this effort can degrade performance, administrators must explicitly enable this feature. ### Enabling or disabling usage metrics using the web console {#efs-metrics-procedure-gui_persistent-storage-csi-aws-efs} To monitor Elastic File System (EFS) volume space consumption through the web console, enable usage metrics by configuring the `ClusterCSIDriver` resource with recursive walk parameters. **Prerequisites** - Access to an OpenShift Container Platform cluster with administrator rights **Procedure** 1. Click **Administration** > **CustomResourceDefinitions**. 2. On the **CustomResourceDefinitions** page next to the **Name** dropdown box, type `clustercsidriver`. 3. Click **CRD ClusterCSIDriver**. 4. Click the **YAML** tab. 5. Under `spec.aws.efsVolumeMetrics.state`: - **Enable** metrics: Set the value to `RecursiveWalk`. RecursiveWalk\` indicates that volume metrics collection in the AWS EFS CSI Driver is performed by recursively walking through the files in the volume. - **Disable** metrics: set the value to `Disabled`. ```yaml {title="Example ClusterCSIDriver efs.csi.aws.com YAML file"} spec: driverConfig: driverType: AWS aws: efsVolumeMetrics: state: RecursiveWalk recursiveWalk: refreshPeriodMinutes: 100 fsRateLimit: 10 ``` 6. Optional: To define how the recursive walk operates, you can also set the following fields: - `refreshPeriodMinutes`: Specifies the refresh frequency for volume metrics in minutes. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 240 minutes. The valid range is 1 to 43,200 minutes. - `fsRateLimit`: Defines the rate limit for processing volume metrics in goroutines per file system. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 5 goroutines. The valid range is 1 to 100 goroutines. 7. Click **Save**. ### Enabling or disabling usage metrics using the CLI {#efs-metrics-procedure-cli_persistent-storage-csi-aws-efs} To monitor Elastic File System (EFS) volume space consumption through the command-line interface, enable usage metrics by configuring the `ClusterCSIDriver` resource with recursive walk parameters. You can also disable usage metrics as needed. **Prerequisites** - Access to an OpenShift Container Platform cluster with administrator rights - Access to OpenShift Container Platform CLI (`oc`) **Procedure** 1. Edit `ClusterCSIDriver` by running the following command: ```terminal $ oc edit clustercsidriver efs.csi.aws.com ``` 2. Under `spec.aws.efsVolumeMetrics.state`: - **Enable** metrics: Set the value to `RecursiveWalk`. `RecursiveWalk` indicates that volume metrics collection in the AWS EFS CSI Driver is performed by recursively walking through the files in the volume. - **Disable** metrics: set the value to `Disabled`. ```yaml {title="Example ClusterCSIDriver efs.csi.aws.com YAML file"} spec: driverConfig: driverType: AWS aws: efsVolumeMetrics: state: RecursiveWalk recursiveWalk: refreshPeriodMinutes: 100 fsRateLimit: 10 ``` 3. Optional: To define how the recursive walk operates, you can also set the following fields: - `refreshPeriodMinutes`: Specifies the refresh frequency for volume metrics in minutes. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 240 minutes. The valid range is 1 to 43,200 minutes. - `fsRateLimit`: Defines the rate limit for processing volume metrics in goroutines per file system. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 5 goroutines. The valid range is 1 to 100 goroutines. 4. Save the changes to the `efs.csi.aws.com` object. ## Amazon Elastic File Storage troubleshooting {#efs-troubleshooting_persistent-storage-csi-aws-efs} To diagnose and resolve AWS Elastic File System (EFS) issues, use these troubleshooting steps including gathering logs, checking operator status, and verifying network connectivity. The following information provides guidance on how to troubleshoot issues with Amazon Elastic File Storage (Amazon EFS): - The AWS EFS Operator and CSI driver run in namespace `openshift-cluster-csi-drivers`. - To initiate gathering of logs of the AWS EFS Operator and CSI driver, run the following command: ```terminal $ oc adm must-gather [must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:125f183d13601537ff15b3239df95d47f0a604da2847b561151fedd699f5e3a5 [must-gather ] OUT namespace/openshift-must-gather-xm4wq created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-2bd8x created [must-gather ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:125f183d13601537ff15b3239df95d47f0a604da2847b561151fedd699f5e3a5 created ``` - To show AWS EFS Operator errors, view the `ClusterCSIDriver` status: ```terminal $ oc get clustercsidriver efs.csi.aws.com -o yaml ``` - If a volume cannot be mounted to a pod (as shown in the output of the following command): ```terminal $ oc describe pod ... Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m13s default-scheduler Successfully assigned default/efs-app to ip-10-0-135-94.ec2.internal Warning FailedMount 13s kubelet MountVolume.SetUp failed for volume "pvc-d7c097e6-67ec-4fae-b968-7e7056796449" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Warning FailedMount 10s kubelet Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage kube-api-access-9j477]: timed out waiting for the condition ``` The text `error: code = DeadlineExceeded desc = context deadline exceeded` is a warning message indicating that the volume is not mounted. This error is frequently caused by AWS dropping packets between an OpenShift Container Platform node and Amazon EFS. Check that the following are correct: - AWS firewall and Security Groups - Networking: port number and IP addresses ## Uninstalling the AWS EFS CSI Driver Operator {#persistent-storage-csi-olm-operator-uninstall_persistent-storage-csi-aws-efs} To remove the AWS EFS Container Storage Interface (CSI) Driver Operator and free cluster resources, uninstall the operator after stopping applications and deleting persistent volumes (PVs). All EFS PVs are inaccessible after uninstalling the AWS EFS CSI Driver Operator (a Red Hat operator). > [!NOTE] > Before you can destroy a cluster (`openshift-install destroy cluster`), you must delete the EFS volume in AWS. An OpenShift Container Platform cluster cannot be destroyed when there is an EFS volume that uses the cluster’s VPC. Amazon does not allow deletion of such a VPC. **Prerequisites** - Access to the OpenShift Container Platform web console. **Procedure** 1. Log in to the web console. 2. Stop all applications that use AWS EFS PVs. 3. Delete all AWS EFS PVs: 1. Click **Storage** → **PersistentVolumeClaims**. 2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click **Delete PersistentVolumeClaims**. 4. Uninstall the AWS EFS CSI driver: > [!NOTE] > Before you can uninstall the Operator, you must remove the CSI driver first. 1. Click **Administration** → **CustomResourceDefinitions** → **ClusterCSIDriver**. 2. On the **Instances** tab, for **{provisioner}**, on the far left side, click the drop-down menu, and then click **Delete ClusterCSIDriver**. 3. When prompted, click **Delete**. 5. Uninstall the AWS EFS CSI Operator: 1. Click **Ecosystem** → **Installed Operators**. 2. On the **Installed Operators** page, scroll or type AWS EFS CSI into the **Search by name** box to find the Operator, and then click it. 3. On the upper, right of the **Installed Operators > Operator details** page, click **Actions** → **Uninstall Operator**. 4. When prompted on the **Uninstall Operator** window, click the **Uninstall** button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually. After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the **Installed Operators** section of the web console. **Additional resources** {._additional-resources} - [Configuring CSI volumes](/openshift-docs-markdown/storage/container_storage_interface/persistent-storage-csi#persistent-storage-csi)