Providing sensitive data to pods by using an external secrets store

As an alternative to using secret objects to provide sensitive information, such as passwords and user names, to applications, you can use an external secret management system to store the information. You can then use the Secrets Store CSI Driver Operator to access the information and mount the secret content as a pod volume. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

About the Secrets Store CSI Driver Operator

To store and manage your secrets securely, configure the Secrets Store CSI Driver Operator to mount secrets from an external secret management system, such as Azure Key Vault, by using a provider plugin. Applications can then use the secret, but the secret does not persist on the system after pod termination.

Secret objects are stored with Base64 encoding. etcd provides encryption at rest for these secrets, but when secrets are retrieved, they are decrypted and presented to the user. If role-based access control is not configured properly on your cluster, anyone with API or etcd access can retrieve or modify a secret. Additionally, anyone who is authorized to create a pod in a namespace can use that access to read any secret in that namespace.

The Secrets Store CSI Driver Operator, secrets-store.csi.k8s.io, enables OpenShift Container Platform to mount multiple secrets, keys, and certificates stored in enterprise-grade external secrets stores into pods as a volume. The Secrets Store CSI Driver Operator communicates with the provider using gRPC to fetch the mount contents from the specified external secrets store. After the volume is attached, the data in it is mounted into the container’s file system. Secrets store volumes are mounted in-line.

For more information about CSI inline volumes, see "CSI inline ephemeral volumes".

Familiarity with persistent storage and configuring CSI volumes is recommended when working with a CSI driver. For more information, see "Understanding persistent storage" and "Configuring CSI volumes".

Additional resources

Secrets store providers

You can store sensitive information needed by your applications in an external secret management system and use the Secrets Store CSI Driver Operator to mount the secret content as a pod volume. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

The Secrets Store CSI Driver Operator has been tested with the following secrets store providers:

AWS Secrets Manager
AWS Systems Manager Parameter Store
Azure Key Vault
Google Secret Manager
HashiCorp Vault

warning

IBM Z supports only HashiCorp Vault.

note

Red Hat does not test all factors associated with third-party secrets store provider functionality. For more information about third-party support, see the "Red Hat third-party support policy".

Automatic rotation

To maintain synchronization with your external secret provider, the Secrets Store CSI Driver Operator automatically rotates secret content in mounted volumes. The Secrets Store CSI Driver Operator uses this process to ensure that updates in the external store are automatically reflected in your pods and secrets.

If you enabled synchronization of mounted content as Kubernetes secrets, the Kubernetes secrets are also rotated.

Applications consuming the secret data must watch for updates to the secrets.

Install the Secrets Store CSI driver

To enable OpenShift Container Platform to mount secrets from external secret management systems, install the Secrets Store CSI Driver Operator and create a ClusterCSIDriver instance.

Prerequisites

Access to the OpenShift Container Platform web console.
Administrator access to the cluster.

Procedure

Install the Secrets Store CSI Driver Operator:
1. Log in to the web console.
2. Click Ecosystem → Software Catalog.
3. Locate the Secrets Store CSI Driver Operator by typing "Secrets Store CSI" in the filter box.
4. Click the Secrets Store CSI Driver Operator button.
5. On the Secrets Store CSI Driver Operator page, click Install.
6. On the Install Operator page, ensure that:
  - All namespaces on the cluster (default) is selected.
  - Installed Namespace is set to openshift-cluster-csi-drivers.
7. Click Install. After the installation finishes, the Secrets Store CSI Driver Operator is listed in the Installed Operators section of the web console.
Create the ClusterCSIDriver instance for the driver (secrets-store.csi.k8s.io):
1. Click Administration → CustomResourceDefinitions → ClusterCSIDriver.
2. On the Instances tab, click Create ClusterCSIDriver. Use the following YAML file:
```
apiVersion: operator.openshift.io/v1
kind: ClusterCSIDriver
metadata:
    name: secrets-store.csi.k8s.io
spec:
  managementState: Managed
```
3. Click Create.

Mount secrets from an external secrets store to a CSI volume

After installing the Secrets Store CSI Driver Operator, you can mount secrets from your external secret store. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

The Secrets Store CSI Driver Operator has been tested with the following secrets store providers:

AWS Secrets Manager
AWS Systems Manager Parameter Store
Azure Key Vault
Google Secret Manager
HashiCorp Vault

Mount secrets from AWS Secrets Manager

You can use the Secrets Store CSI Driver Operator to mount secrets from AWS Secrets Manager external secrets store to a Container Storage Interface (CSI) volume in OpenShift Container Platform. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the jq tool.
You have extracted and prepared the ccoctl utility.
You have installed the cluster on Amazon Web Services (AWS) and the cluster uses AWS Security Token Service (STS).
You have installed the Secrets Store CSI Driver Operator. For more information, see "Installing the Secrets Store CSI driver".
You have configured AWS Secrets Manager to store the required secrets.

Procedure

Install the AWS Secrets Manager provider:

Create a YAML file by using the following example configuration:

warning

The AWS Secrets Manager provider for the Secrets Store CSI driver is an upstream provider.

This configuration is modified from the configuration provided in the upstream AWS documentation so that it works properly with OpenShift Container Platform. Changes to this configuration might impact functionality.

Example aws-provider.yaml file
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-aws
  namespace: openshift-cluster-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-aws-cluster-role
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-aws-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
subjects:
- kind: ServiceAccount
  name: csi-secrets-store-provider-aws
  namespace: openshift-cluster-csi-drivers
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: openshift-cluster-csi-drivers
  name: csi-secrets-store-provider-aws
  labels:
    app: csi-secrets-store-provider-aws
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-aws
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-aws
    spec:
      serviceAccountName: csi-secrets-store-provider-aws
      hostNetwork: false
      containers:
        - name: provider-aws-installer
          image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-50-g5b4aca1-2023.06.09.21.19
          imagePullPolicy: Always
          args:
              - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: HostToContainer
      tolerations:
      - operator: Exists
      volumes:
        - name: providervol
          hostPath:
            path: "/etc/kubernetes/secrets-store-csi-providers"
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
      nodeSelector:
        kubernetes.io/os: linux

Grant privileged access to the csi-secrets-store-provider-aws service account by running the following command:

$ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-aws -n openshift-cluster-csi-drivers

Create the provider resources by running the following command:
```
$ oc apply -f aws-provider.yaml
```

Grant the read permission to the service account for the AWS secret object:

Create a directory to contain the credentials request by running the following command:
```
$ mkdir <aws_creds_directory_name>
```

Create a YAML file that defines the CredentialsRequest resource configuration. See the following example configuration:

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: aws-creds-request
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - "secretsmanager:GetSecretValue"
      - "secretsmanager:DescribeSecret"
      effect: Allow
      resource: "arn:*:secretsmanager:*:*:secret:testSecret-??????"
  secretRef:
    name: aws-creds
    namespace: my-namespace
  serviceAccountNames:
  - <service_account_name>

Retrieve the OpenID Connect (OIDC) provider by running the following command:
```
$ oc get --raw=/.well-known/openid-configuration | jq -r '.issuer'
```
Example output
```
https://<oidc_provider_name>
```
Copy the OIDC provider name <oidc_provider_name> from the output to use in the next step.

Use the ccoctl tool to process the credentials request by running the following command:

$ ccoctl aws create-iam-roles \
    --name my-role --region=<aws_region> \
    --credentials-requests-dir=<aws_creds_dir_name> \
    --identity-provider-arn arn:aws:iam::<aws_account_id>:oidc-provider/<oidc_provider_name> --output-dir=<output_dir_name>

Example output
2023/05/15 18:10:34 Role arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds created
2023/05/15 18:10:34 Saved credentials configuration to: credrequests-ccoctl-output/manifests/my-namespace-aws-creds-credentials.yaml
2023/05/15 18:10:35 Updated Role policy for Role my-role-my-namespace-aws-creds

Copy the <aws_role_arn> from the output to use in the next step. For example, arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds.

Bind the service account with the role ARN by running the following command:

$ oc annotate -n my-namespace sa/aws-provider eks.amazonaws.com/role-arn="<aws_role_arn>"

Create a secret provider class to define your secrets store provider:
1. Create a YAML file that defines the SecretProviderClass object:
  Example secret-provider-class-aws.yaml
```
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-aws-provider
  namespace: my-namespace
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "testSecret"
        objectType: "secretsmanager"
```
  where:
  
  metadata.name
  
  Specifies the name for the secret provider class.
  
  metadata.namespace
  
  Specifies the namespace for the secret provider class.
  
  spec.provider
  
  Specifies the provider as aws.
  
  spec.parameters
  
  Specifies the provider-specific configuration parameters.
2. Create the SecretProviderClass object by running the following command:
```
$ oc create -f secret-provider-class-aws.yaml
```

Create a deployment to use this secret provider class:

Create a YAML file that defines the Deployment object:

Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-aws-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-storage
  template:
    metadata:
      labels:
        app: my-storage
    spec:
      serviceAccountName: aws-provider
      containers:
      - name: busybox
        image: k8s.gcr.io/e2e-test-images/busybox:1.29
        command:
          - "/bin/sleep"
          - "10000"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "my-aws-provider"

where:

metadata.name: Specifies the name for the deployment.
metadata.namespace: Specifies the namespace for the deployment. This must be the same namespace as the secret provider class.
spec.template.spec.volumes.csi.volumeAttributes.secretProviderClass: Specifies the name of the secret provider class.

Create the Deployment object by running the following command:
```
$ oc create -f deployment.yaml
```

Verification

Verify that you can access the secrets from AWS Secrets Manager in the pod volume mount:
1. List the secrets in the pod mount by running the following command:
```
$ oc exec my-aws-deployment-<hash> -n my-namespace -- ls /mnt/secrets-store/
```
  Example output
```
testSecret
```
2. View a secret in the pod mount by running the following command:
```
$ oc exec my-aws-deployment-<hash> -n my-namespace -- cat /mnt/secrets-store/testSecret
```
  Example output
```
<secret_value>
```

Mount secrets from AWS Systems Manager Parameter Store

You can use the Secrets Store CSI Driver Operator to mount secrets from AWS Systems Manager Parameter Store external secrets store to a Container Storage Interface (CSI) volume in OpenShift Container Platform. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the jq tool.
You have extracted and prepared the ccoctl utility.
You have installed the cluster on Amazon Web Services (AWS) and the cluster uses AWS Security Token Service (STS).
You have installed the Secrets Store CSI Driver Operator. For more information, see "Installing the Secrets Store CSI driver".
You have configured AWS Systems Manager Parameter Store to store the required secrets.

Procedure

Install the AWS Systems Manager Parameter Store provider:

Create a YAML file by using the following example configuration:

warning

The AWS Systems Manager Parameter Store provider for the Secrets Store CSI driver is an upstream provider.

Example aws-provider.yaml file
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-aws
  namespace: openshift-cluster-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-aws-cluster-role
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-aws-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
subjects:
- kind: ServiceAccount
  name: csi-secrets-store-provider-aws
  namespace: openshift-cluster-csi-drivers
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: openshift-cluster-csi-drivers
  name: csi-secrets-store-provider-aws
  labels:
    app: csi-secrets-store-provider-aws
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-aws
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-aws
    spec:
      serviceAccountName: csi-secrets-store-provider-aws
      hostNetwork: false
      containers:
        - name: provider-aws-installer
          image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-50-g5b4aca1-2023.06.09.21.19
          imagePullPolicy: Always
          args:
              - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: HostToContainer
      tolerations:
      - operator: Exists
      volumes:
        - name: providervol
          hostPath:
            path: "/etc/kubernetes/secrets-store-csi-providers"
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
      nodeSelector:
        kubernetes.io/os: linux

Grant privileged access to the csi-secrets-store-provider-aws service account by running the following command:

$ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-aws -n openshift-cluster-csi-drivers

Create the provider resources by running the following command:
```
$ oc apply -f aws-provider.yaml
```

Grant the read permission to the service account for the AWS secret object:

Create a directory to contain the credentials request by running the following command:
```
$ mkdir <aws_creds_directory_name>
```

Create a YAML file that defines the CredentialsRequest resource configuration. See the following example configuration:

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: aws-creds-request
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - "ssm:GetParameter"
      - "ssm:GetParameters"
      effect: Allow
      resource: "arn:*:ssm:*:*:parameter/testParameter*"
  secretRef:
    name: aws-creds
    namespace: my-namespace
  serviceAccountNames:
  - <service_account_name>

Retrieve the OpenID Connect (OIDC) provider by running the following command:
```
$ oc get --raw=/.well-known/openid-configuration | jq -r '.issuer'
```
Example output
```
https://<oidc_provider_name>
```
Copy the OIDC provider name <oidc_provider_name> from the output to use in the next step.

Use the ccoctl tool to process the credentials request by running the following command:

$ ccoctl aws create-iam-roles \
    --name my-role --region=<aws_region> \
    --credentials-requests-dir=<aws_creds_dir_name> \
    --identity-provider-arn arn:aws:iam::<aws_account_id>:oidc-provider/<oidc_provider_name> --output-dir=<output_dir_name>

Example output
2023/05/15 18:10:34 Role arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds created
2023/05/15 18:10:34 Saved credentials configuration to: credrequests-ccoctl-output/manifests/my-namespace-aws-creds-credentials.yaml
2023/05/15 18:10:35 Updated Role policy for Role my-role-my-namespace-aws-creds

Copy the <aws_role_arn> from the output to use in the next step. For example, arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds.

Bind the service account with the role ARN by running the following command:

$ oc annotate -n my-namespace sa/aws-provider eks.amazonaws.com/role-arn="<aws_role_arn>"

Create a secret provider class to define your secrets store provider:
1. Create a YAML file that defines the SecretProviderClass object:
  Example secret-provider-class-aws.yaml
```
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-aws-provider
  namespace: my-namespace
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "testParameter"
        objectType: "ssmparameter"
```
  where:
  
  metadata.name
  
  Specifies the name for the secret provider class.
  
  metadata.namespace
  
  Specifies the namespace for the secret provider class.
  
  spec.provider
  
  Specifies the provider as aws.
  
  spec.parameters
  
  Specifies the provider-specific configuration parameters.
2. Create the SecretProviderClass object by running the following command:
```
$ oc create -f secret-provider-class-aws.yaml
```

Create a deployment to use this secret provider class:

Create a YAML file that defines the Deployment object:

Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-aws-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-storage
  template:
    metadata:
      labels:
        app: my-storage
    spec:
      serviceAccountName: aws-provider
      containers:
      - name: busybox
        image: k8s.gcr.io/e2e-test-images/busybox:1.29
        command:
          - "/bin/sleep"
          - "10000"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "my-aws-provider"

where:

metadata.name: Specifies the name for the deployment.
metadata.namespace: Specifies the namespace for the deployment. This must be the same namespace as the secret provider class.
spec.template.spec.volumes.csi.volumeAttributes.secretProviderClass: Specifies the name of the secret provider class.

Create the Deployment object by running the following command:
```
$ oc create -f deployment.yaml
```

Verification

Verify that you can access the secrets from AWS Systems Manager Parameter Store in the pod volume mount:
1. List the secrets in the pod mount by running the following command:
```
$ oc exec my-aws-deployment-<hash> -n my-namespace -- ls /mnt/secrets-store/
```
  Example output
```
testParameter
```
2. View a secret in the pod mount by running the following command:
```
$ oc exec my-aws-deployment-<hash> -n my-namespace -- cat /mnt/secrets-store/testSecret
```
  Example output
```
<secret_value>
```

Mount secrets from Azure Key Vault

You can use the Secrets Store CSI Driver Operator to mount secrets from Microsoft Azure Key Vault to a Container Storage Interface (CSI) volume in OpenShift Container Platform. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

Prerequisites

Your have installed a cluster on Azure.
You have access to the cluster as a user with the cluster-admin role.
You have installed the Azure CLI (az).
You have installed the Secrets Store CSI Driver Operator. See "Installing the Secrets Store CSI driver" for instructions.
You have configured Azure Key Vault to store the required secrets.

Procedure

Install the Azure Key Vault provider:

Create a YAML file named azure-provider.yaml that defines the ServiceAccount resource configuration. See the following example configuration:

warning

The Azure Key Vault provider for the Secrets Store CSI driver is an upstream provider.

This configuration is modified from the configuration provided in the upstream Azure documentation so that it works properly with OpenShift Container Platform. Changes to this configuration might impact functionality.

Example azure-provider.yaml file
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-azure
  namespace: openshift-cluster-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-azure-cluster-role
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-azure-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-azure-cluster-role
subjects:
- kind: ServiceAccount
  name: csi-secrets-store-provider-azure
  namespace: openshift-cluster-csi-drivers
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: openshift-cluster-csi-drivers
  name: csi-secrets-store-provider-azure
  labels:
    app: csi-secrets-store-provider-azure
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-azure
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-azure
    spec:
      serviceAccountName: csi-secrets-store-provider-azure
      hostNetwork: true
      containers:
        - name: provider-azure-installer
          image: mcr.microsoft.com/oss/azure/secrets-store/provider-azure:v1.4.1
          imagePullPolicy: IfNotPresent
          args:
            - --endpoint=unix:///provider/azure.sock
            - --construct-pem-chain=true
            - --healthz-port=8989
            - --healthz-path=/healthz
            - --healthz-timeout=5s
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8989
            failureThreshold: 3
            initialDelaySeconds: 5
            timeoutSeconds: 10
            periodSeconds: 30
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 0
            capabilities:
              drop:
              - ALL
          volumeMounts:
            - mountPath: "/provider"
              name: providervol
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      volumes:
        - name: providervol
          hostPath:
            path: "/var/run/secrets-store-csi-providers"
      tolerations:
      - operator: Exists
      nodeSelector:
        kubernetes.io/os: linux

Grant privileged access to the csi-secrets-store-provider-azure service account by running the following command:

$ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-azure -n openshift-cluster-csi-drivers

Create the provider resources by running the following command:
```
$ oc apply -f azure-provider.yaml
```

Create a service principal to access the key vault:

Set the service principal client secret as an environment variable by running the following command:

$ SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --name https://$KEYVAULT_NAME --query 'password' -otsv)"

Set the service principal client ID as an environment variable by running the following command:

$ SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name https://$KEYVAULT_NAME --query '[0].appId' -otsv)"

Create a generic secret with the service principal client secret and ID by running the following command:

$ oc create secret generic secrets-store-creds -n my-namespace --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET}

Apply the secrets-store.csi.k8s.io/used=true label to allow the provider to find this nodePublishSecretRef secret:
```
$ oc -n my-namespace label secret secrets-store-creds secrets-store.csi.k8s.io/used=true
```

Create a secret provider class to define your secrets store provider:

Create a YAML file that defines the SecretProviderClass object:

Example secret-provider-class-azure.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-azure-provider
  namespace: my-namespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: "kvname"
    objects: |
      array:
        - |
          objectName: secret1
          objectType: secret
    tenantId: "tid"

where:

metadata.name: Specifies the name for the secret provider class.
metadata.namespace: Specifies the namespace for the secret provider class.
spec.provider: Specifies the provider as azure.
spec.parameters: Specifies the provider-specific configuration parameters.

Create the SecretProviderClass object by running the following command:
```
$ oc create -f secret-provider-class-azure.yaml
```

Create a deployment to use this secret provider class:

Create a YAML file that defines the Deployment object:

Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-azure-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-storage
  template:
    metadata:
      labels:
        app: my-storage
    spec:
      containers:
      - name: busybox
        image: k8s.gcr.io/e2e-test-images/busybox:1.29
        command:
          - "/bin/sleep"
          - "10000"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "my-azure-provider"
            nodePublishSecretRef:
              name: secrets-store-creds

where:

metadata.name: Specifies the name for the deployment.
metadata.namespace: Specifies the namespace for the deployment. This must be the same namespace as the secret provider class.
spec.template.spec.volumes.csi.volumeAttributes.secretProviderClass: Specifies the name of the secret provider class.
spec.template.spec.volumes.csi.nodePublishSecretRef.name: Specifies the name of the Kubernetes secret that contains the service principal credentials to access Azure Key Vault.

Create the Deployment object by running the following command:
```
$ oc create -f deployment.yaml
```

Verification

Verify that you can access the secrets from Azure Key Vault in the pod volume mount:
1. List the secrets in the pod mount by running the following command:
```
$ oc exec my-azure-deployment-<hash> -n my-namespace -- ls /mnt/secrets-store/
```
  Example output
```
secret1
```
2. View a secret in the pod mount by running the following command:
```
$ oc exec my-azure-deployment-<hash> -n my-namespace -- cat /mnt/secrets-store/secret1
```
  Example output
```
my-secret-value
```

Mount secrets from Google Secret Manager

You can use the Secrets Store CSI Driver Operator to mount secrets from Google Secret Manager to a Container Storage Interface (CSI) volume in OpenShift Container Platform. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

Prerequisites

Your have installed a cluster on Google Cloud.
You have access to the cluster as a user with the cluster-admin role.
You have installed the Secrets Store CSI Driver Operator. See "Installing the Secrets Store CSI driver" for instructions.
You have configured Google Secret Manager to store the required secrets.
You have created a service account key named key.json from your Google Cloud service account.

Procedure

Install the Google Secret Manager provider:

Create a YAML file Create a YAML file named gcp-provider.yaml that defines the ServiceAccount resource configuration. See the following example configuration:

Example gcp-provider.yaml file
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-gcp
  namespace: openshift-cluster-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-gcp-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-gcp-role
subjects:
  - kind: ServiceAccount
    name: csi-secrets-store-provider-gcp
    namespace: openshift-cluster-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-gcp-role
rules:
  - apiGroups:
      - ""
    resources:
      - serviceaccounts/token
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - serviceaccounts
    verbs:
      - get
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-secrets-store-provider-gcp
  namespace: openshift-cluster-csi-drivers
  labels:
    app: csi-secrets-store-provider-gcp
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-gcp
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-gcp
    spec:
      serviceAccountName: csi-secrets-store-provider-gcp
      initContainers:
      - name: chown-provider-mount
        image: busybox
        command:
        - chown
        - "1000:1000"
        - /etc/kubernetes/secrets-store-csi-providers
        volumeMounts:
        - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
          name: providervol
        securityContext:
          privileged: true
      hostNetwork: false
      hostPID: false
      hostIPC: false
      containers:
        - name: provider
          image: us-docker.pkg.dev/secretmanager-csi/secrets-store-csi-driver-provider-gcp/plugin@sha256:a493a78bbb4ebce5f5de15acdccc6f4d19486eae9aa4fa529bb60ac112dd6650
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          env:
            - name: TARGET_DIR
              value: "/etc/kubernetes/secrets-store-csi-providers"
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
              mountPropagation: None
              readOnly: false
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /live
              port: 8095
            initialDelaySeconds: 5
            timeoutSeconds: 10
            periodSeconds: 30
      volumes:
        - name: providervol
          hostPath:
            path: /etc/kubernetes/secrets-store-csi-providers
      tolerations:
        - key: kubernetes.io/arch
          operator: Equal
          value: amd64
          effect: NoSchedule
      nodeSelector:
        kubernetes.io/os: linux

Grant privileged access to the csi-secrets-store-provider-gcp service account by running the following command:

$ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-gcp -n openshift-cluster-csi-drivers

Create the provider resources by running the following command:
```
$ oc apply -f gcp-provider.yaml
```

Grant a read permission to the Google Secret Manager secret:
1. Create a new project by running the following command:
```
$ oc new-project my-namespace
```
2. Label the my-namespace namespace for pod security admission by running the following command:
```
$ oc label ns my-namespace security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite
```
3. Create a service account for the pod deployment:
```
$ oc create serviceaccount my-service-account --namespace=my-namespace
```
4. Create a generic secret from the key.json file by running the following command:
```
$ oc create secret generic secrets-store-creds -n my-namespace --from-file=key.json
```
  You created this key.json file from the Google Secret Manager.
5. Apply the secrets-store.csi.k8s.io/used=true label to allow the provider to find this nodePublishSecretRef secret:
```
$ oc -n my-namespace label secret secrets-store-creds secrets-store.csi.k8s.io/used=true
```
Create a secret provider class to define your secrets store provider:
1. Create a YAML file that defines the SecretProviderClass object:
  Example secret-provider-class-gcp.yaml
```
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-gcp-provider
  namespace: my-namespace
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/my-project/secrets/testsecret1/versions/1"
        path: "testsecret1.txt"
```
  where:
  
  metadata.name
  
  Specifies the name for the secret provider class.
  
  metadata.namespace
  
  Specifies the namespace for the secret provider class.
  
  spec.provider
  
  Specifies the provider as gcp.
  
  spec.parameters
  
  Specifies the provider-specific configuration parameters.
2. Create the SecretProviderClass object by running the following command:
```
$ oc create -f secret-provider-class-gcp.yaml
```

Create a deployment to use this secret provider class:

Create a YAML file that defines the Deployment object:

Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-gcp-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-storage
  template:
    metadata:
      labels:
        app: my-storage
    spec:
      serviceAccountName: my-service-account
      containers:
      - name: busybox
        image: k8s.gcr.io/e2e-test-images/busybox:1.29
        command:
          - "/bin/sleep"
          - "10000"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "my-gcp-provider"
            nodePublishSecretRef:
              name: secrets-store-creds

where:

metadata.name: Specifies the name for the deployment.
metadata.namespace: Specifies the namespace for the deployment. This must be the same namespace as the secret provider class.
spec.template.spec.serviceAccountName: Specifies the service account you created.
spec.template.spec.volumes.csi.volumeAttributes.secretProviderClass: Specifies the name of the secret provider class.
spec.template.spec.volumes.csi.nodePublishSecretRef.name: Specifies the name of the Kubernetes secret that contains the service principal credentials to access Google Secret Manager.

Create the Deployment object by running the following command:
```
$ oc create -f deployment.yaml
```

Verification

Verify that you can access the secrets from Google Secret Manager in the pod volume mount:
1. List the secrets in the pod mount by running the following command:
```
$ oc exec my-gcp-deployment-<hash> -n my-namespace -- ls /mnt/secrets-store/
```
  Example output
```
testsecret1
```
2. View a secret in the pod mount by running the following command:
```
$ oc exec my-gcp-deployment-<hash> -n my-namespace -- cat /mnt/secrets-store/testsecret1
```
  Example output
```
<secret_value>
```

Mount secrets from HashiCorp Vault

You can use the Secrets Store CSI Driver Operator to mount secrets from HashiCorp Vault to a Container Storage Interface (CSI) volume in OpenShift Container Platform. Using an external secret store protects information that you do not want developers to have and can be more secure than secret objects.

warning

Mounting secrets from HashiCorp Vault by using the Secrets Store CSI Driver Operator has been tested with the following cloud providers:

Amazon Web Services (AWS)
Microsoft Azure

Other cloud providers might work, but have not been tested yet. Additional cloud providers might be tested in the future.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the Secrets Store CSI Driver Operator. See "Installing the Secrets Store CSI driver" for instructions.
You have installed Helm.

Procedure

Add the HashiCorp Helm repository by running the following command:
```
$ helm repo add hashicorp https://helm.releases.hashicorp.com
```
Update all repositories to ensure that Helm is aware of the latest versions by running the following command:
```
$ helm repo update
```

Install the HashiCorp Vault provider:

Create a new project for Vault by running the following command:
```
$ oc new-project vault
```

Label the vault namespace for pod security admission by running the following command:

$ oc label ns vault security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite

Grant privileged access to the vault service account by running the following command:
```
$ oc adm policy add-scc-to-user privileged -z vault -n vault
```
Grant privileged access to the vault-csi-provider service account by running the following command:
```
$ oc adm policy add-scc-to-user privileged -z vault-csi-provider -n vault
```
If you are using IBM Z, create a vault-license secret for HashiCorp Vault Enterprise:
1. Create a vault-license.yaml file with the following content:
```
apiVersion: v1
data:
  license: <license-string>
kind: Secret
metadata:
  name: vault-license
  namespace: vault
type: Opaque
```
2. Create the secret by running the following command:
```
$ oc create -f vault-license.yaml
```

Deploy HashiCorp Vault by running the following command:

$ helm install vault hashicorp/vault --namespace=vault \
  --set "server.dev.enabled=true" \
  --set "injector.enabled=false" \
  --set "csi.enabled=true" \
  --set "global.openshift=true" \
  --set "injector.agentImage.repository=docker.io/hashicorp/vault" \
  --set "server.image.repository=docker.io/hashicorp/vault" \
  --set "csi.image.repository=docker.io/hashicorp/vault-csi-provider" \
  --set "csi.agent.image.repository=docker.io/hashicorp/vault" \
  --set "csi.daemonSet.providersDir=/var/run/secrets-store-csi-providers"

If you are using IBM Z, you must use Vault Enterprise images and the Vault license secret. Run the following command instead:

$ helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set server.dev.enabled=true \
  --set server.image.repository="docker.io/hashicorp/vault-enterprise" \
  --set server.image.tag="1.21-ent" \
  --set server.enterpriseLicense.secretName="vault-license" \
  --set server.logLevel=debug \
  --set server.serviceAccount.name="vault" \
  --set "injector.enabled=false" \
  --set global.openshift=true \
  --set csi.enabled=true \
  --set "csi.daemonSet.providersDir=/var/run/secrets-store-csi-providers" \
  --set csi.image.repository="registry.connect.redhat.com/hashicorp/vault-csi-provider" \
  --set csi.image.tag="1.7.1-ubi" \
  --set csi.agent.enabled=false

Patch the vault-csi-driver daemon set to set the securityContext to privileged by running the following command:

$ oc patch daemonset -n vault vault-csi-provider --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": true} }]'

Verify that the vault-csi-provider pods have started properly by running the following command:

$ oc get pods -n vault

Example output
NAME                       READY   STATUS    RESTARTS   AGE
vault-0                    1/1     Running   0          24m
vault-csi-provider-87rgw   1/2     Running   0          5s
vault-csi-provider-bd6hp   1/2     Running   0          4s
vault-csi-provider-smlv7   1/2     Running   0          5s

Configure HashiCorp Vault to store the required secrets:

Create a secret by running the following command:

$ oc exec vault-0 --namespace=vault -- vault kv put secret/example1 testSecret1=my-secret-value

Verify that the secret is readable at the path secret/example1 by running the following command:

$ oc exec vault-0 --namespace=vault -- vault kv get secret/example1

Example output
= Secret Path =
secret/data/example1

======= Metadata =======
Key                Value
---                -----
created_time       2024-04-05T07:05:16.713911211Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======= Data =======
Key            Value
---            -----
testSecret1    my-secret-value

Configure Vault to use Kubernetes authentication:

Enable the Kubernetes auth method by running the following command:

$ oc exec vault-0 --namespace=vault -- vault auth enable kubernetes

Example output
Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes auth method:

Set the token reviewer as an environment variable by running the following command:

$ TOKEN_REVIEWER_JWT="$(oc exec vault-0 --namespace=vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

Set the Kubernetes service IP address as an environment variable by running the following command:

$ KUBERNETES_SERVICE_IP="$(oc get svc kubernetes --namespace=default -o go-template="{{ .spec.clusterIP }}")"

Update the Kubernetes auth method by running the following command:

$ oc exec -i vault-0 --namespace=vault -- vault write auth/kubernetes/config \
  issuer="https://kubernetes.default.svc.cluster.local" \
  token_reviewer_jwt="${TOKEN_REVIEWER_JWT}" \
  kubernetes_host="https://${KUBERNETES_SERVICE_IP}:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Example output
Success! Data written to: auth/kubernetes/config

Create a policy for the application by running the following command:

$ oc exec -i vault-0 --namespace=vault -- vault policy write csi -<<EOF
  path "secret/data/*" {
  capabilities = ["read"]
  }
  EOF

Example output
Success! Uploaded policy: csi

Create an authentication role to access the application by running the following command:

$ oc exec -i vault-0 --namespace=vault -- vault write auth/kubernetes/role/csi \
  bound_service_account_names=default \
  bound_service_account_namespaces=default,test-ns,negative-test-ns,my-namespace \
  policies=csi \
  ttl=20m

Example output
Success! Data written to: auth/kubernetes/role/csi

Create a secret provider class to define your secrets store provider:
1. Create a YAML file that defines the SecretProviderClass object:
  Example secret-provider-class-vault.yaml
```
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-vault-provider
  namespace: my-namespace
spec:
  provider: vault
  parameters:
    roleName: "csi"
    vaultAddress: "http://vault.vault:8200"
    objects:  |
      - secretPath: "secret/data/example1"
        objectName: "testSecret1"
        secretKey: "testSecret1"
```
  where:
  
  metadata.name
  
  Specifies the name for the secret provider class.
  
  metadata.namespace
  
  Specifies the namespace for the secret provider class.
  
  spec.provider
  
  Specifies the provider as vault.
  
  spec.parameters
  
  Specifies the provider-specific configuration parameters.
2. Create the SecretProviderClass object by running the following command:
```
$ oc create -f secret-provider-class-vault.yaml
```

Create a deployment to use this secret provider class:

Create a YAML file that defines the Deployment object:

Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  namespace: my-namespace
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - image: registry.k8s.io/e2e-test-images/busybox:1.29-4
        name: busybox
        imagePullPolicy: IfNotPresent
        command:
        - "/bin/sleep"
        - "10000"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "my-vault-provider"

where:

metadata.name: Specifies the name for the deployment.
metadata.namespace: Specifies the namespace for the deployment. This must be the same namespace as the secret provider class.
spec.template.spec.volumes.csi.volumeAttributes.secretProviderClass: Specifies the name of the secret provider class.

Create the Deployment object by running the following command:
```
$ oc create -f deployment.yaml
```

Verification

Verify that all of the vault pods are running properly by running the following command:

$ oc get pods -n vault

Example output
NAME                       READY   STATUS    RESTARTS   AGE
vault-0                    1/1     Running   0          43m
vault-csi-provider-87rgw   2/2     Running   0          19m
vault-csi-provider-bd6hp   2/2     Running   0          19m
vault-csi-provider-smlv7   2/2     Running   0          19m

Verify that all of the secrets-store-csi-driver pods are running by running the following command:

$ oc get pods -n openshift-cluster-csi-drivers | grep -E "secrets"

Example output
secrets-store-csi-driver-node-46d2g                  3/3     Running   0             45m
secrets-store-csi-driver-node-d2jjn                  3/3     Running   0             45m
secrets-store-csi-driver-node-drmt4                  3/3     Running   0             45m
secrets-store-csi-driver-node-j2wlt                  3/3     Running   0             45m
secrets-store-csi-driver-node-v9xv4                  3/3     Running   0             45m
secrets-store-csi-driver-node-vlz28                  3/3     Running   0             45m
secrets-store-csi-driver-operator-84bd699478-fpxrw   1/1     Running   0             47m

Verify that you can access the secrets from your HashiCorp Vault in the pod volume mount:
1. List the secrets in the pod mount by running the following command:
```
$ oc exec busybox-deployment-<hash> -n my-namespace -- ls /mnt/secrets-store/
```
  Example output
```
testSecret1
```
2. View a secret in the pod mount by running the following command:
```
$ oc exec busybox-deployment-<hash> -n my-namespace -- cat /mnt/secrets-store/testSecret1
```
  Example output
```
my-secret-value
```

Enable synchronization of mounted content as Kubernetes secrets

You can enable a synchronization process that creates secret objects from the content on a mounted volume. Using secrets protects information that you do not want developers to have.

An example where you might want to enable synchronization is to use an environment variable in your deployment to reference the Kubernetes secret.

warning

Do not enable synchronization if you do not want to store your secrets on your OpenShift Container Platform cluster and in etcd. Enable this functionality only if you require it, such as when you want to use environment variables to refer to the secret.

If you enable synchronization, the secrets from the mounted volume are synchronized as Kubernetes secrets after you start a pod that mounts the secrets.

The synchronized Kubernetes secret is deleted when all pods that mounted the content are deleted.

Prerequisites

You have installed the Secrets Store CSI Driver Operator.
You have installed a secrets store provider.
You have created the secret provider class.
You have access to the cluster as a user with the cluster-admin role.

Procedure

Edit the SecretProviderClass resource by running the following command:
```
$ oc edit secretproviderclass my-azure-provider
```
Replace my-azure-provider with the name of your secret provider class.

Add the secretsObjects section with the configuration for the synchronized Kubernetes secrets:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-azure-provider
  namespace: my-namespace
spec:
  provider: azure
  secretObjects:
    - secretName: tlssecret
      type: kubernetes.io/tls
      labels:
        environment: "test"
      data:
        - objectName: tlskey
          key: tls.key
        - objectName: tlscrt
          key: tls.crt
  parameters:
    usePodIdentity: "false"
    keyvaultName: "kvname"
    objects:  |
      array:
        - |
          objectName: tlskey
          objectType: secret
        - |
          objectName: tlscrt
          objectType: secret
    tenantId: "tid"

where:

spec.secretObjects: Specifies the configuration for synchronized Kubernetes secrets.
spec.secretObjects.secretname: Specifies the name of the Kubernetes Secret object to create.
spec.secretObjects.type: Specifies the type of Kubernetes Secret object to create. For example, Opaque or kubernetes.io/tls.
spec.secretObjects.data.object.name: Specifies the object name or alias of the mounted content to synchronize.
spec.secretObjects.data.object.key: Specifies the data field from the specified objectName to populate the Kubernetes secret with.

Save the file to apply the changes.

View the status of secrets in the pod volume mount

You can view detailed information of the secrets, including the versions, in the pod volume mount. You can use this information to help you confirm that secrets from your external store are active within the pod environment.

The Secrets Store CSI Driver Operator creates a SecretProviderClassPodStatus resource in the same namespace as the pod. You can review this resource to see detailed information, including versions, about the secrets in the pod volume mount.

Prerequisites

You have installed the Secrets Store CSI Driver Operator.
You have installed a secrets store provider.
You have created the secret provider class.
You have deployed a pod that mounts a volume from the Secrets Store CSI Driver Operator.
You have access to the cluster as a user with the cluster-admin role.

Procedure

View detailed information about the secrets in a pod volume mount by running the following command:

$ oc get secretproviderclasspodstatus <secret_provider_class_pod_status_name> -o yaml (1)

Replace <secret_provider_class_pod_status_name> with the name of the secret provider class pod status object is in the format of <pod_name>-<namespace>-<secret_provider_class_name>.

Example output
...
status:
  mounted: true
  objects:
  - id: secret/tlscrt
    version: f352293b97da4fa18d96a9528534cb33
  - id: secret/tlskey
    version: 02534bc3d5df481cb138f8b2a13951ef
  podName: busybox-<hash>
  secretProviderClassName: my-azure-provider
  targetPath: /var/lib/kubelet/pods/f0d49c1e-c87a-4beb-888f-37798456a3e7/volumes/kubernetes.io~csi/secrets-store-inline/mount

Uninstall the Secrets Store CSI Driver Operator

To remove the Secrets Store CSI Driver Operator and free cluster resources, uninstall the Operator after stopping applications and removing the CSI driver.

Prerequisites

Access to the OpenShift Container Platform web console.
Administrator access to the cluster.

Procedure

Stop all application pods that use the secrets-store.csi.k8s.io provider.
Remove any third-party provider plug-in for your chosen secret store.
Remove the Container Storage Interface (CSI) driver and associated manifests:
1. Click Administration → CustomResourceDefinitions → ClusterCSIDriver.
2. On the Instances tab, for secrets-store.csi.k8s.io, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
3. When prompted, click Delete.
Verify that the CSI driver pods are no longer running.
Uninstall the Secrets Store CSI Driver Operator:

note
Before you can uninstall the Operator, you must remove the CSI driver first.
1. Click Ecosystem → Installed Operators.
2. On the Installed Operators page, scroll or type "Secrets Store CSI" into the Search by name box to find the Operator, and then click it.
3. On the upper, right of the Installed Operators > Operator details page, click Actions → Uninstall Operator.
4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually. After uninstalling, the Secrets Store CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

Additional resources

About the Secrets Store CSI Driver Operator​

Secrets store providers​

Automatic rotation​

Install the Secrets Store CSI driver​

Mount secrets from an external secrets store to a CSI volume​

Mount secrets from AWS Secrets Manager​

Mount secrets from AWS Systems Manager Parameter Store​

Mount secrets from Azure Key Vault​

Mount secrets from Google Secret Manager​

Mount secrets from HashiCorp Vault​

Enable synchronization of mounted content as Kubernetes secrets​

View the status of secrets in the pod volume mount​

Uninstall the Secrets Store CSI Driver Operator​

About the Secrets Store CSI Driver Operator

Secrets store providers

Automatic rotation

Install the Secrets Store CSI driver

Mount secrets from an external secrets store to a CSI volume

Mount secrets from AWS Secrets Manager

Mount secrets from AWS Systems Manager Parameter Store

Mount secrets from Azure Key Vault

Mount secrets from Google Secret Manager

Mount secrets from HashiCorp Vault

Enable synchronization of mounted content as Kubernetes secrets

View the status of secrets in the pod volume mount

Uninstall the Secrets Store CSI Driver Operator