--- title: Configuring PCI passthrough --- # Configuring PCI passthrough {#virt-configuring-pci-passthrough} The Peripheral Component Interconnect (PCI) passthrough feature enables you to access and manage hardware devices from a virtual machine (VM). When PCI passthrough is configured, the PCI devices function as if they were physically attached to the guest operating system. Cluster administrators can expose and manage host devices that are permitted to be used in the cluster by using the `oc` command-line interface (CLI). > [!IMPORTANT] > For `vfio-pci` to allocate a PCI device, no other kernel driver can manage that device. If a driver already manages the device, you must add the specific kernel module to a blocklist. > > Adding a kernel module to a blocklist makes all devices handled by that module unavailable to the host. The following example shows a `MachineConfig` CR that adds the `enic` network driver to a blocklist by creating a configuration file in `/etc/modprobe.d/` and adding kernel arguments: ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 100-blacklist-enic spec: config: ignition: version: 3.4.0 storage: files: - contents: source: data:,blacklist%20enic%0A mode: 420 overwrite: true path: /etc/modprobe.d/blacklist-enic.conf kernelArguments: - enic.blacklist=1 - rd.driver.blacklist=enic ``` ## Preparing nodes for GPU passthrough {#virt-preparing-nodes-for-gpu-passthrough} You can prevent GPU operands from deploying on worker nodes that you designated for GPU passthrough. ### Preventing NVIDIA GPU operands from deploying on nodes {#virt-preventing-nvidia-gpu-operands-from-deploying-on-nodes_virt-configuring-pci-passthrough} If you use the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/openshift/contents.html) in your cluster, you can apply the `nvidia.com/gpu.deploy.operands=false` label to nodes that you do not want to configure for GPU or vGPU operands. This prevents the creation of the pods that configure GPU or vGPU operands and terminates existing pods. **Prerequisites** - The OpenShift CLI (`oc`) is installed. **Procedure** - Label the node by running the following command: ```terminal $ oc label node nvidia.com/gpu.deploy.operands=false ``` where: `` : Specifies the name of a node where you do not want to install the NVIDIA GPU operands. **Verification** 1. Verify that the label was added to the node by running the following command: ```terminal $ oc describe node ``` 2. Optional: If GPU operands were previously deployed on the node, verify their removal. 1. Check the status of the pods in the `nvidia-gpu-operator` namespace by running the following command: ```terminal $ oc get pods -n nvidia-gpu-operator ``` Example output: ```terminal NAME READY STATUS RESTARTS AGE gpu-operator-59469b8c5c-hw9wj 1/1 Running 0 8d nvidia-sandbox-validator-7hx98 1/1 Running 0 8d nvidia-sandbox-validator-hdb7p 1/1 Running 0 8d nvidia-sandbox-validator-kxwj7 1/1 Terminating 0 9d nvidia-vfio-manager-7w9fs 1/1 Running 0 8d nvidia-vfio-manager-866pz 1/1 Running 0 8d nvidia-vfio-manager-zqtck 1/1 Terminating 0 9d ``` 2. Monitor the pod status until the pods with `Terminating` status are removed: ```terminal $ oc get pods -n nvidia-gpu-operator ``` Example output: ```terminal NAME READY STATUS RESTARTS AGE gpu-operator-59469b8c5c-hw9wj 1/1 Running 0 8d nvidia-sandbox-validator-7hx98 1/1 Running 0 8d nvidia-sandbox-validator-hdb7p 1/1 Running 0 8d nvidia-vfio-manager-7w9fs 1/1 Running 0 8d nvidia-vfio-manager-866pz 1/1 Running 0 8d ``` ## Preparing host devices for PCI passthrough {#virt-preparing-host-devices-for-pci-passthrough} ### About preparing a host device for PCI passthrough {#virt-about_pci-passthrough_virt-configuring-pci-passthrough} To prepare a host device for PCI passthrough by using the CLI, create a `MachineConfig` object and add kernel arguments to enable the Input-Output Memory Management Unit (IOMMU). Bind the PCI device to the Virtual Function I/O (VFIO) driver and then expose it in the cluster by editing the `permittedHostDevices` field of the `HyperConverged` custom resource (CR). The `permittedHostDevices` list is empty when you first install the OpenShift Virtualization Operator. To remove a PCI host device from the cluster by using the CLI, delete the PCI device information from the `HyperConverged` CR. ### Adding kernel arguments to enable the IOMMU driver {#virt-adding-kernel-arguments-enable-IOMMU_virt-configuring-pci-passthrough} You must enable the Input-Output Memory Management Unit (IOMMU) driver before you can configure mediated devices. To enable the IOMMU driver in the kernel, create the `MachineConfig` object and add the kernel arguments. **Prerequisites** - You have cluster administrator permissions. - Your CPU hardware is Intel or AMD. > [!NOTE] > Enabling IOMMU is not required on `s390x` architecture. - You enabled Intel Virtualization Technology for Directed I/O extensions or AMD IOMMU in the BIOS. - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Create a `MachineConfig` object that identifies the kernel argument. The following example shows a kernel argument for an Intel CPU. ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 100-worker-iommu spec: config: ignition: version: 3.2.0 kernelArguments: - intel_iommu=on # ... ``` - `metadata.labels.machineconfiguration.openshift.io/role` specifies that the new kernel argument is applied only to worker nodes. - `metadata.name` specifies the ranking of this kernel argument (100) among the machine configs and its purpose. If you have an AMD CPU, specify the kernel argument as `amd_iommu=on`. - `spec.kernelArguments` specifies the kernel argument as `intel_iommu` for an Intel CPU. 2. Create the new `MachineConfig` object: ```terminal $ oc create -f 100-worker-kernel-arg-iommu.yaml ``` **Verification** 1. Verify that the new `MachineConfig` object was added by entering the following command and observing the output: ```terminal $ oc get MachineConfig ``` Example output: ```terminal NAME IGNITIONVERSION AGE 00-master 3.5.0 164m 00-worker 3.5.0 164m 01-master-container-runtime 3.5.0 164m 01-master-kubelet 3.5.0 164m 01-worker-container-runtime 3.5.0 164m 01-worker-kubelet 3.5.0 164m 100-master-chrony-configuration 3.5.0 169m 100-master-set-core-user-password 3.5.0 169m 100-worker-chrony-configuration 3.5.0 169m 100-worker-iommu 3.5.0 14s ``` 2. Verify that IOMMU is enabled at the operating system (OS) level by entering the following command: ```terminal $ dmesg | grep -i iommu ``` - If IOMMU is enabled, output is displayed as shown in the following example: Example output: ```terminal Intel: [ 0.000000] DMAR: Intel(R) IOMMU Driver AMD: [ 0.000000] AMD-Vi: IOMMU Initialized ``` ### Binding PCI devices to the VFIO driver {#virt-binding-devices-vfio-driver_virt-configuring-pci-passthrough} To bind PCI devices to the VFIO (Virtual Function I/O) driver, obtain the values for the vendor ID and the device ID from each device and create a list with the values. Add this list to the `MachineConfig` object. The `MachineConfig` Operator generates the `/etc/modprobe.d/vfio.conf` on the nodes with the PCI devices, and binds the PCI devices to the VFIO driver. **Prerequisites** - You added kernel arguments to enable IOMMU for the CPU. > [!NOTE] > Enabling IOMMU is not required on `s390x` architecture. - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Run the `lspci` command with the name of the GPU accelerator to obtain the vendor ID and the device ID for the PCI device. > [!NOTE] > NVIDIA GPU is supported on `x86` and `aarch64` architectures, Intel QAT is supported on `x86` architecture, and IBM(R) Spyre is supported on `s390x` architecture. ```terminal $ lspci -nnv | grep -i ``` Valid values for `` are `nvidia`, `qat`, and `spyre`. Example output: ```terminal 02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1) ``` 2. Create a Butane config file, `100-worker-vfiopci.bu`, binding the PCI device to the VFIO driver. > [!NOTE] > The [Butane version](https://coreos.github.io/butane/specs/) you specify in the config file should match the OpenShift Container Platform version and always ends in `0`. For example, `4.22.0`. See "Creating machine configs with Butane" for information about Butane. Example: ```yaml variant: openshift version: 4.22.0 metadata: name: 100-worker-vfiopci labels: machineconfiguration.openshift.io/role: worker storage: files: - path: /etc/modprobe.d/vfio.conf mode: 0644 overwrite: true contents: inline: | options vfio-pci ids=: - path: /etc/modules-load.d/vfio-pci.conf mode: 0644 overwrite: true contents: inline: vfio-pci ``` - `metadata.labels.machineconfiguration.openshift.io/role: worker` specifies that the new kernel argument is applied only to compute nodes. - `storage.files.contents.inline`, where the path is `/etc/modprobe.d/vfio.conf`, specifies the previously determined hexadecimal vendor ID and device ID values to bind a device to the VFIO driver. You can add a list of multiple devices with their vendor and device information. - `storage.files.path`, where the `contents.inline` is `vfio-pci`, specifies the file that loads the `vfio-pci` kernel module on the compute nodes. 3. Use Butane to generate a `MachineConfig` object file, `100-worker-vfiopci.yaml`, containing the configuration to be delivered to the compute nodes: ```terminal $ butane 100-worker-vfiopci.bu -o 100-worker-vfiopci.yaml ``` 4. Apply the `MachineConfig` object to the compute nodes: ```terminal $ oc apply -f 100-worker-vfiopci.yaml ``` 5. Verify that the `MachineConfig` object was added. ```terminal $ oc get MachineConfig ``` Example output: ```terminal NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.5.0 25h 00-worker d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.5.0 25h 01-master-container-runtime d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.5.0 25h 01-master-kubelet d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.5.0 25h 01-worker-container-runtime d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.5.0 25h 01-worker-kubelet d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.5.0 25h 100-worker-iommu 3.5.0 30s 100-worker-vfiopci-configuration 3.5.0 30s ``` **Verification** - Verify that the VFIO driver is loaded. ```terminal $ lspci -nnk -d : ``` The output confirms that the VFIO driver is being used. Example output: ``` 04:00.0 3D controller [0302]: NVIDIA Corporation GP102GL [Tesla P40] [10de:1eb8] (rev a1) Subsystem: NVIDIA Corporation Device [10de:1eb8] Kernel driver in use: vfio-pci Kernel modules: nouveau ``` ### Exposing PCI host devices in the cluster using the CLI {#virt-exposing-pci-device-in-cluster-cli_virt-configuring-pci-passthrough} To expose PCI host devices in the cluster, add details about the PCI devices to the `spec.permittedHostDevices.pciHostDevices` array of the `HyperConverged` custom resource (CR). **Prerequisites** - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Edit the `HyperConverged` CR in your default editor by running the following command: ```terminal $ oc edit hyperconvergeds.v1beta1.hco.kubevirt.io kubevirt-hyperconverged -n openshift-cnv ``` 2. Add the PCI device information to the `spec.permittedHostDevices.pciHostDevices` array. Example configuration file: ```yaml apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: permittedHostDevices: pciHostDevices: - pciDeviceSelector: "10DE:1DB6" resourceName: "nvidia.com/GV100GL_Tesla_V100" - pciDeviceSelector: "10DE:1EB8" resourceName: "nvidia.com/TU104GL_Tesla_T4" - pciDeviceSelector: "8086:6F54" resourceName: "intel.com/qat" externalResourceProvider: true # ... ``` - `spec.permittedHostDevices` specifies the host devices that are permitted to be used in the cluster. - `spec.permittedHostDevices.pciHostDevices` specifies the list of PCI devices available on the node. - `spec.permittedHostDevices.pciHostDevices.pciDeviceSelector` specifies the vendor ID and the device ID required to identify the PCI device. - `spec.permittedHostDevices.pciHostDevices.resourceName` specifies the name of a PCI host device. - `spec.permittedHostDevices.pciHostDevices.externalResourceProvider` is an optional setting. Setting this field to `true` indicates that the resource is provided by an external device plugin. OpenShift Virtualization allows the usage of this device in the cluster but leaves the allocation and monitoring to an external device plugin. > [!NOTE] > The above example snippet shows two PCI host devices that are named `nvidia.com/GV100GL_Tesla_V100` and `nvidia.com/TU104GL_Tesla_T4` added to the list of permitted host devices in the `HyperConverged` CR. These devices have been tested and verified to work with OpenShift Virtualization. Example configuration file for an IBM(R) Spyre device on `s390x` architecture: ```yaml apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: permittedHostDevices: pciHostDevices: - pciDeviceSelector: "1014:06a8" resourceName: "ibm.com/spyre" # ... ``` 3. Save your changes and exit the editor. **Verification** - Verify that the PCI host devices were added to the node by running the following command. The example output shows that there is one device each associated with the `nvidia.com/GV100GL_Tesla_V100`, `nvidia.com/TU104GL_Tesla_T4`, and `intel.com/qat` resource names. ```terminal $ oc describe node ``` Example output: ```terminal Capacity: cpu: 64 devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 915128Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 131395264Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 1 intel.com/qat: 1 pods: 250 Allocatable: cpu: 63500m devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 863623130526 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 130244288Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 1 intel.com/qat: 1 pods: 250 ``` > [!NOTE] > When using an IBM(R) Spyre device on `s390x` architecture, the allocated device is shown as follows: `ibm.com/spyre:         1`. ### Removing PCI host devices from the cluster using the CLI {#virt-removing-pci-device-from-cluster_virt-configuring-pci-passthrough} To remove a PCI host device from the cluster, delete the information for that device from the `HyperConverged` custom resource (CR). **Prerequisites** - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Edit the `HyperConverged` CR in your default editor by running the following command: ```terminal $ oc edit hyperconvergeds.v1beta1.hco.kubevirt.io kubevirt-hyperconverged -n openshift-cnv ``` 2. Remove the PCI device information from the `spec.permittedHostDevices.pciHostDevices` array by deleting the `pciDeviceSelector`, `resourceName` and `externalResourceProvider` (if applicable), fields for the appropriate device. In this example, the user deletes the `nvidia.com/TU104GL_Tesla_T4`. Example configuration file: ```yaml apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: permittedHostDevices: pciHostDevices: - pciDeviceSelector: "10DE:1DB6" resourceName: "nvidia.com/GV100GL_Tesla_V100" # ... ``` 3. Save your changes and exit the editor. **Verification** - Verify that you removed the PCI host device from the node by running the following command. The example output shows that there are zero devices associated with the `nvidia.com/TU104GL_Tesla_T4` resource name. ```terminal $ oc describe node ``` Example output: ```terminal Capacity: cpu: 64 devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 915128Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 131395264Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 0 pods: 250 Allocatable: cpu: 63500m devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 863623130526 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 130244288Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 0 pods: 250 ``` ## Configuring virtual machines for PCI passthrough {#virt-configuring-vms-for-pci-passthrough} After the PCI devices have been added to the cluster, you can assign them to virtual machines. The PCI devices are now available as if they are physically connected to the virtual machines. ### Assigning a PCI device to a virtual machine {#virt-assigning-pci-device-virtual-machine_virt-configuring-pci-passthrough} When a PCI device is available in a cluster, you can assign it to a virtual machine and enable PCI passthrough. **Procedure** - Assign the PCI device to a virtual machine as a host device. Example: ```yaml apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: domain: devices: hostDevices: - deviceName: nvidia.com/TU104GL_Tesla_T4 name: hostdevices1 ``` - `spec.template.spec.domain.devices.hostDevices.deviceName` specifies the name of the PCI device that is permitted on the cluster as a host device. The virtual machine can access this host device. When using an IBM(R) Spyre device on `s390x` architecture, specify `ibm.com/spyre:`. **Verification** - Use the following command to verify that the host device is available from the virtual machine. ```terminal $ lspci -nnk | grep ``` Valid values for `` are `nvidia`, `qat`, and `spyre`. Example output: ```terminal $ 02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1) ``` **Additional resources** {._additional-resources} - [Enabling Intel VT-X and AMD-V Virtualization Hardware Extensions in BIOS](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-troubleshooting-enabling_intel_vt_x_and_amd_v_virtualization_hardware_extensions_in_bios) - [Managing file permissions](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/assembly_managing-file-permissions_configuring-basic-system-settings) - [Machine Config Overview](/openshift-docs-markdown/machine_configuration/index#machine-config-overview) - [IBM(R) Spyre Accelerator User’s Guide](https://www.ibm.com/docs/en/systems-hardware/linuxone/9175-ML1?topic=library-spyre-accelerator-users-guide)