Configuring an RDMA subsystem for SR-IOV

Remote Direct Memory Access (RDMA) allows direct memory access between two systems without involving the operating system of either system. You can configure an RDMA Container Network Interface (CNI) on Single Root I/O Virtualization (SR-IOV) to enable high-performance, low-latency communication between containers. When you combine RDMA with SR-IOV, you provide a mechanism to expose hardware counters of Mellanox Ethernet devices for use inside Data Plane Development Kit (DPDK) applications.

Configure SR-IOV RDMA CNI

Configure an RDMA CNI on SR-IOV.

note

This procedure applies only to Mellanox devices.

Prerequisites

You have installed the OpenShift CLI (oc).
You have access to the cluster as a user with the cluster-admin role.
You have installed the SR-IOV Network Operator.

Procedure

Create an SriovNetworkPoolConfig CR and save it as sriov-nw-pool.yaml, as shown in the following example:

Example SriovNetworkPoolConfig CR
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkPoolConfig
metadata:
  name: worker
  namespace: openshift-sriov-network-operator
spec:
  maxUnavailable: 1
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  rdmaMode: <rdma_mode>

spec.rdmaMode: Specifies the RDMA network namespace mode. Set to exclusive.

Create the SriovNetworkPoolConfig resource by running the following command:
```
$ oc create -f sriov-nw-pool.yaml
```

Create an SriovNetworkNodePolicy CR and save it as sriov-node-policy.yaml, as shown in the following example:

Example SriovNetworkNodePolicy CR
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: sriov-nic-pf1
  namespace: openshift-sriov-network-operator
spec:
  deviceType: netdevice
  isRdma: <is_rdma>
  nicSelector:
    pfNames: ["ens3f0np0"]
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  numVfs: 4
  priority: 99
  resourceName: sriov_nic_pf1

spec.isRdma: Specifies whether to activate RDMA mode. Set to true.

Create the SriovNetworkNodePolicy resource by running the following command:
```
$ oc create -f sriov-node-policy.yaml
```

Create an SriovNetwork CR and save it as sriov-network.yaml, as shown in the following example:

Example SriovNetwork CR
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: sriov-nic-pf1
  namespace: openshift-sriov-network-operator
spec:
  networkNamespace: sriov-tests
  resourceName: sriov_nic_pf1
    ipam: |-
  metaPlugins: |
    {
      "type": "<plugin_type>"
    }

spec.metaPlugins.type: Specifies the type of meta plugin. Set to rdma to create the RDMA plugin.

Create the SriovNetwork resource by running the following command:
```
$ oc create -f sriov-network.yaml
```

Verification

Create a Pod CR and save it as sriov-test-pod.yaml, as shown in the following example:

Example runtime configuration
apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: |-
      [
        {
          "name": "net1",
          "mac": "20:04:0f:f1:88:01",
          "ips": ["192.168.10.1/24", "2001::1/64"]
        }
      ]
spec:
  containers:
  - name: sample-container
    image: <image>
    imagePullPolicy: IfNotPresent
    command: ["sleep", "infinity"]

Create the test pod by running the following command:
```
$ oc create -f sriov-test-pod.yaml
```
Log in to the test pod by running the following command:
```
$ oc rsh testpod1 -n sriov-tests
```

Verify that the path to the hw-counters directory exists by running the following command:

$ ls /sys/bus/pci/devices/${PCIDEVICE_OPENSHIFT_IO_SRIOV_NIC_PF1}/infiniband/*/ports/1/hw_counters/

Example output
duplicate_request       out_of_buffer req_cqe_flush_error           resp_cqe_flush_error        roce_adp_retrans        roce_slow_restart_trans
implied_nak_seq_err     out_of_sequence req_remote_access_errors    resp_local_length_error     roce_adp_retrans_to     rx_atomic_requests
lifespan                packet_seq_err req_remote_invalid_request   resp_remote_access_errors   roce_slow_restart       rx_read_requests
local_ack_timeout_err  req_cqe_error resp_cqe_error                 rnr_nak_retry_err           roce_slow_restart_cnps  rx_write_requests

Configure SR-IOV RDMA CNI​

Configure SR-IOV RDMA CNI