--- title: Collecting data for Red Hat Support --- # Collecting data for Red Hat Support {#virt-collecting-virt-data} When you submit a support case to Red Hat Support, it is helpful to provide debugging information for OpenShift Container Platform and OpenShift Virtualization. You can use the following tools to collect debugging information: must-gather tool : The `must-gather` tool collects diagnostic information, including resource definitions and service logs. Prometheus : Prometheus is a time-series database and a rule evaluation engine for metrics. Prometheus sends alerts to Alertmanager for processing. Alertmanager : The Alertmanager service handles alerts received from Prometheus. The Alertmanager is also responsible for sending the alerts to external notification systems. ## Collecting data about your environment {#virt-collecting-data-about-your-environment_virt-collecting-virt-data} Collecting data about your environment minimizes the time required to analyze and determine the root cause. **Prerequisites** - You have set the retention time for Prometheus metrics data to a minimum of seven days. - You have configured the Alertmanager to capture relevant alerts and to send alert notifications to a dedicated mailbox so that they can be viewed and persisted outside the cluster. - You have recorded the exact number of affected nodes and virtual machines. **Procedure** 1. Collect must-gather data for the cluster. 2. Collect must-gather data for Red Hat OpenShift Data Foundation, if necessary. 3. Collect must-gather data for OpenShift Virtualization. 4. Collect Prometheus metrics for the cluster. ## Collecting data about virtual machines {#virt-collecting-data-about-vms_virt-collecting-virt-data} Collecting data about malfunctioning virtual machines (VMs) minimizes the time required to analyze and determine the root cause. **Prerequisites** - For Linux VMs, you have installed the latest QEMU guest agent. - For Windows VMs, you have: - Recorded the Windows patch update details. - Installed the latest VirtIO drivers. - Installed the latest QEMU guest agent. - If Remote Desktop Protocol (RDP) is enabled, you have connected by using the desktop viewer to determine whether there is a problem with the connection software. **Procedure** 1. Collect must-gather data for the VMs using the `/usr/bin/gather` script. 2. Collect screenshots of VMs that have crashed before you restart them. 3. Collect memory dumps from VMs before remediation attempts. 4. Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network. ## Using the must-gather tool for OpenShift Virtualization {#virt-using-virt-must-gather_virt-collecting-virt-data} You can collect data about OpenShift Virtualization resources by running the `must-gather` command with the OpenShift Virtualization image. The default data collection includes information about the following resources: - OpenShift Virtualization Operator namespaces, including child objects - OpenShift Virtualization custom resource definitions - Namespaces that contain virtual machines - Basic virtual machine definitions You can add optional environment details and scripts to the `must-gather` command to collect additional information. Use these environment variables and scripts to collect data about specific VMs, images, or instance types. **Prerequisites** - You have installed the OpenShift CLI (`oc`). **Procedure** - Run the `must-gather` command to collect data about OpenShift Virtualization: ```terminal $ oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.22.6 \ -- /usr/bin/gather ``` > [!NOTE] > You can also collect `must-gather` logs for all Operators and products on your cluster by running following command: > > ```terminal > $ oc adm must-gather --all-images > ``` 1. Run the following command to modify the number of processes running in parallel when collecting `must-gather` data: ```terminal $ oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.22.6 \ -- PROS= /usr/bin/gather ``` `PROS` defines the number of parallel processes running to collect data. The default number of processes is 5. Increasing the number of processes may result in faster data collection, but uses more resources. Increasing the maximum number of parallel processes is not recommended. 2. Run the following command to collect detailed information for a specific VM in a specific namespace: ```terminal $ oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.22.6 \ -- NS= VM= /usr/bin/gather --vms_details ``` `NS` is the environment variable for `namespace`. It is mandatory when using the `VM` environment variable. 3. Run the following command to collect image, image-stream, and image-stream-tags information from the cluster: ```terminal $ oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.22.6 \ /usr/bin/gather --images ``` 4. Run the following command to collect information about instance types from the cluster: ```terminal $ oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.22.6 \ /usr/bin/gather --instancetypes ``` ### must-gather tool options {#virt-must-gather-options_virt-collecting-virt-data} To troubleshoot complex issues and collect specific data beyond the default logs, add optional parameters to the `must-gather` command when gathering information from your cluster. You can specify a combination of scripts and environment variables for the following options: - Collecting detailed virtual machine (VM) information from a namespace - Collecting detailed information about specified VMs - Collecting image, image-stream, and image-stream-tags information - Limiting the maximum number of parallel processes used by the `must-gather` tool #### Environment variables {#must-gather-environment-variables_virt-collecting-virt-data} You can specify environment variables for a compatible script. `NS=` : Collect virtual machine information, including `virt-launcher` pod details, from the namespace that you specify. The `VirtualMachine` and `VirtualMachineInstance` CR data is collected for all namespaces. `VM=` : Collect details about a particular virtual machine. To use this option, you must also specify a namespace by using the `NS` environment variable. `PROS=` : Modify the maximum number of parallel processes that the `must-gather` tool uses. The default value is `5`. > [!IMPORTANT] > Using too many parallel processes can cause performance issues. Increasing the maximum number of parallel processes is not recommended. #### Scripts {#must-gather-scripts_virt-collecting-virt-data} Each script is compatible only with certain environment variable combinations. `/usr/bin/gather` : Use the default `must-gather` script, which collects cluster data from all namespaces and includes only basic VM information. This script is compatible only with the `PROS` variable. `/usr/bin/gather --vms_details` : Collect VM log files, VM definitions, control-plane logs, and namespaces that belong to OpenShift Virtualization resources. Specifying namespaces includes their child objects. If you use this parameter without specifying a namespace or VM, the `must-gather` tool collects this data for all VMs in the cluster. This script is compatible with all environment variables, but you must specify a namespace if you use the `VM` variable. `/usr/bin/gather --images` : Collect image, image-stream, and image-stream-tags custom resource information. This script is compatible only with the `PROS` variable. `/usr/bin/gather --instancetypes` : Collect instance types information. This information is not currently collected by default; you can, however, optionally collect it. #### Usage and examples {#usage-and-examples_virt-collecting-virt-data} You can run a script by itself or with one or more compatible environment variables. ```terminal {title="must-gather syntax with optional parameters"} $ oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.22.6 \ -- ``` **Compatible parameters** | Script | Compatible environment variable | | --- | --- | | `/usr/bin/gather` | \* `PROS=` | | `/usr/bin/gather --vms_details` | \* For a namespace: `NS=`
\* For a VM: `VM= NS=`
\* `PROS=` | | `/usr/bin/gather --images` | \* `PROS=` | ## Generating a VM memory dump {#virt-generating-a-vm-memory-dump_virt-collecting-virt-data} When a virtual machine (VM) terminates unexpectedly, you can use the `virtctl memory-dump` to generate a memory dump command to output a VM memory dump and save it on a persistent volume claim (PVC). Afterwards, you can analyze the memory dump to diagnose and troubleshoot issues on the VM. **Procedure** 1. Optional: You have an existing PVC on which you want to save the memory dump. - The PVC volume mode must be `FileSystem`. - The PVC must be large enough to contain the memory dump. The formula for calculating the PVC size is `(VMMemorySize + 100Mi) * (1 + FileSystemOverhead)`, where `100Mi` is the memory dump overhead, and `FileSystemOverhead` is defined in the `HCO` object. 2. Create a memory dump of the required VM: - If you have an existing PVC selected on which you want to save the memory dump: ```terminal $ virtctl memory-dump get --claim-name= ``` - If you want to create a new PVC for the memory dump: ```terminal $ virtctl memory-dump get --claim-name= --create-claim ``` 3. Download the memory dump: ```terminal $ virtctl memory-dump download --output= ``` 4. Attach the memory dump to a Red Hat Support case. Alternatively, you can inspect the memory dump, for example by using the volatility3 tool. 5. Optional: Remove the memory dump: ```terminal $ virtctl memory-dump remove ``` ## Self-service Technical Supportability Review {#about-self-service-tsr_virt-collecting-virt-data} You can use the self-service Technical Supportability Review (TSR) on the Red Hat Customer Portal to validate your cluster configuration against Red Hat common practices. > [!NOTE] > The `must-gather` tool collects diagnostic information about your cluster, including resource definitions, service logs, and configuration data. For more information, see "Gathering data about your cluster" in the OpenShift Container Platform documentation. The self-service TSR uses AI to evaluate your cluster’s `must-gather` data and provides a prioritized executive summary of recommendations. This serves as a starting point to help you identify and resolve potential issues before they impact your environment. The TSR performs hundreds of checks across the OpenShift Container Platform platform, including OpenShift Virtualization. Coverage is continually expanding. ### When to use the self-service TSR tool {#when-to-use-self-service-tsr_virt-collecting-virt-data} Integrating the self-service TSR into your regular operational workflow can be helpful in the following scenarios: Routine benchmarking : Use the TSR quarterly to benchmark cluster health and plan for routine maintenance activities. Pre-flight checks : Validate your cluster configuration before major structural changes, including upgrades, migrations, and expansions. Critical event preparation : Confirm cluster stability ahead of high-traffic business events, such as seasonal peaks, or operational milestones, such as year-end shutdowns, business continuity drills, and compliance audits. ### How to access the TSR {#how-to-access-tsr_virt-collecting-virt-data} To run a self-service review, upload your cluster’s `must-gather` data to the **Analyze** tab in the **Support** section of the Red Hat Customer Portal. For a direct link, see "Technical Supportability Review with AI tool" in the Additional resources section. The **Analyze** feature generates a prioritized executive summary that identifies your cluster’s top risks and recommends corrective actions. Review the recommendations and implement the suggested corrective actions to address the identified risks. The self-service TSR provides a solid baseline for cluster health. If you need additional guidance or a more comprehensive review, contact your Red Hat account team to arrange an assisted review through a Technical Account Manager (TAM) or Red Hat consultant. An assisted review includes human analysis, deeper coverage, and access to checks that are updated more frequently than the self-service version. **Additional resources** {._additional-resources} - [Technical Supportability Review with AI tool](https://access.redhat.com/support/cases/#/analyze) - [Red Hat Technical Supportability Review with AI: Proactive AI-Driven Cluster Assessments for OpenShift Container Platform](https://access.redhat.com/solutions/7141255) **Additional resources** {._additional-resources} - [VM support overview](/openshift-docs-markdown/virt/support/virt-support-overview#virt-support-overview) - [How to provide log files to Red Hat Support (Red Hat Knowledgebase)](https://access.redhat.com/solutions/2112) - [About OpenShift Container Platform monitoring](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/about_monitoring/about-ocp-monitoring) - [Installing the QEMU guest agent on a Linux VM](/openshift-docs-markdown/virt/managing_vms/virt-installing-qemu-guest-agent#virt-installing-qemu-guest-agent-on-linux-vm_virt-installing-qemu-guest-agent) - [Installing VirtIO drivers from a SATA CD drive on an existing Windows VM](/openshift-docs-markdown/virt/managing_vms/virt-install-virtio-drivers-on-windows-vms#virt-installing-virtio-drivers-existing-windows_virt-install-virtio-drivers-on-windows-vms) - [Connect to the desktop viewer by using the web console](/openshift-docs-markdown/virt/managing_vms/virt-accessing-vm-consoles#virt-connecting-desktop-viewer-web_virt-accessing-vm-consoles) - [Collect memory dumps from VMs](/openshift-docs-markdown/virt/support/virt-collecting-virt-data#virt-generating-a-vm-memory-dump_virt-collecting-virt-data) - [Submitting a support case](/openshift-docs-markdown/support/getting-support#support-submitting-a-case_getting-support) - [Modifying retention time and size for Prometheus metrics data](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/configuring_user_workload_monitoring/storing-and-recording-data-uwm#modifying-retention-time-and-size-for-prometheus-metrics-data_storing-and-recording-data-uwm) - [Configuring the Alertmanager to capture relevant alerts and to send alert notifications to a dedicated mailbox](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/configuring_user_workload_monitoring/storing-and-recording-data-uwm#modifying-retention-time-and-size-for-prometheus-metrics-data_storing-and-recording-data-uwm) - [Modifying retention time and size for Prometheus metrics data](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/configuring_core_platform_monitoring/storing-and-recording-data#modifying-retention-time-and-size-for-prometheus-metrics-data_storing-and-recording-data) - [Configuring alerts and notifications](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/configuring_core_platform_monitoring/configuring-alerts-and-notifications) - [Downloading log files and diagnostic information](https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/latest/html-single/troubleshooting_openshift_data_foundation/index#downloading-log-files-and-diagnostic-information_rhodf) - [Querying metrics for all projects with the monitoring dashboard](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/accessing_metrics/accessing-metrics-as-an-administrator#querying-metrics-for-all-projects-with-mon-dashboard_accessing-metrics-as-an-administrator) - [Installing the latest VirtIO drivers](https://access.redhat.com/solutions/6957701) - [Volatility3 tool](https://github.com/volatilityfoundation/volatility3) - [Customer Support](https://access.redhat.com/support/cases/#/case/list) - [Create issue](https://redhat.atlassian.net/secure/CreateIssue.jspa)