--- title: Troubleshooting hosted control planes --- # Troubleshooting hosted control planes {#hcp-troubleshooting} If you encounter an issue with hosted control planes, you can gather information about the hosted cluster, OpenShift Container Platform, or other components so that you can determine the root cause and take steps to resolve it. ## Gathering information to troubleshoot hosted control planes {#hosted-control-planes-troubleshooting_hcp-troubleshooting} When you need to troubleshoot an issue with hosted clusters, you can gather information by running the `must-gather` command. The command generates output for the management cluster and the hosted cluster. The output for the management cluster contains the following content: - **Cluster-scoped resources:** These resources are node definitions of the management cluster. - **The `hypershift-dump` compressed file:** This file is useful if you need to share the content with other people. - **Namespaced resources:** These resources include all of the objects from the relevant namespaces, such as config maps, services, events, and logs. - **Network logs:** These logs include the OVN northbound and southbound databases and the status for each one. - **Hosted clusters:** This level of output involves all of the resources inside of the hosted cluster. The output for the hosted cluster contains the following content: - **Cluster-scoped resources:** These resources include all of the cluster-wide objects, such as nodes and CRDs. - **Namespaced resources:** These resources include all of the objects from the relevant namespaces, such as config maps, services, events, and logs. Although the output does not contain any secret objects from the cluster, it can contain references to the names of secrets. **Prerequisites** - You must have `cluster-admin` access to the management cluster. - You need the `name` value for the `HostedCluster` resource and the namespace where the CR is deployed. - You must have the `hcp` command-line interface installed. For more information, see "Installing the hosted control planes command-line interface". - You must have the OpenShift CLI (`oc`) installed. - You must ensure that the `kubeconfig` file is loaded and is pointing to the management cluster. **Procedure** - To gather the output for troubleshooting, enter the following command: ```terminal $ oc adm must-gather \ --image=registry.redhat.io/rhacm2/acm-must-gather-rhel9:v2.17 \ /usr/bin/gather hosted-cluster-namespace=HOSTEDCLUSTERNAMESPACE \ hosted-cluster-name=HOSTEDCLUSTERNAME \ --dest-dir=NAME ; tar -cvzf NAME.tgz NAME ``` where: - The `hosted-cluster-namespace=HOSTEDCLUSTERNAMESPACE` parameter is optional. If you do not include it, the command runs as though the hosted cluster is in the default namespace, which is `clusters`. - If you want to save the results of the command to a compressed file, specify the `--dest-dir=NAME` parameter and replace `NAME` with the name of the directory where you want to save the results. **Additional resources** {._additional-resources} - [Installing the hosted control planes command-line interface](/openshift-docs-markdown/hosted_control_planes/hcp-prepare/hcp-cli#hcp-cli) ## Gathering OpenShift Container Platform data for a hosted cluster {#hcp-must-gather-day-2_hcp-troubleshooting} You can gather OpenShift Container Platform debugging information for a hosted cluster by using the multicluster engine Operator web console or by using the CLI. ### Gathering data for a hosted cluster by using the CLI {#hcp-must-gather-cli_hcp-troubleshooting} You can gather OpenShift Container Platform debugging information for a hosted cluster by using the command-line interface (CLI). **Prerequisites** - You must have `cluster-admin` access to the management cluster. - You need the `name` value for the `HostedCluster` resource and the namespace where the CR is deployed. - You must have the `hcp` command-line interface installed. For more information, see "Installing the hosted control planes command-line interface". - You must have the OpenShift CLI (`oc`) installed. - You must ensure that the `kubeconfig` file is loaded and is pointing to the management cluster. **Procedure** 1. Generate the `kubeconfig` file by entering the following command: ```terminal $ hcp create kubeconfig --namespace \ --name > .kubeconfig ``` 2. After you save the `kubeconfig` file, you can access the hosted cluster by entering the following example command: ```terminal $ oc --kubeconfig .kubeconfig get nodes ``` 3. Collect the must-gather information by entering the following command: ```terminal $ oc adm must-gather ``` ### Gathering data for a hosted cluster by using the web console {#hcp-must-gather-console_hcp-troubleshooting} You can gather OpenShift Container Platform debugging information for a hosted cluster by using the multicluster engine Operator web console. **Prerequisites** - You must have `cluster-admin` access to the management cluster. - You need the `name` value for the `HostedCluster` resource and the namespace where the CR is deployed. - You must have the `hcp` command-line interface installed. For more information, see "Installing the hosted control planes command-line interface". - You must have the OpenShift CLI (`oc`) installed. - You must ensure that the `kubeconfig` file is loaded and is pointing to the management cluster. **Procedure** 1. In the web console, select **All Clusters** and select the cluster you want to troubleshoot. 2. In the upper-right corner, select **Download kubeconfig**. 3. Export the downloaded `kubeconfig` file. 4. Collect the must-gather information by entering the following command: ```terminal $ oc adm must-gather ``` ## Entering the must-gather command in a disconnected environment {#hcp-must-gather-dc_hcp-troubleshooting} When you need to troubleshoot an issue in a disconnected environment, you can gather information by running the `must-gather` command. The command generates output for the management cluster and the hosted cluster. **Procedure** 1. In a disconnected environment, mirror the Red Hat Operator catalog images into their mirror registry. For more information, see "Install on disconnected networks". 2. Run the following command to extract logs that reference the image from their mirror registry: ```terminal REGISTRY=registry.example.com:5000 IMAGE=$REGISTRY/rhacm2/acm-must-gather-rhel9:v2.17 $ oc adm must-gather \ --image=$IMAGE /usr/bin/gather \ hosted-cluster-namespace=HOSTEDCLUSTERNAMESPACE \ hosted-cluster-name=HOSTEDCLUSTERNAME \ --dest-dir=./data ``` **Additional resources** {._additional-resources} - [Install on disconnected networks](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/latest/html/clusters/cluster_mce_overview#install-on-disconnected-networks) ## Troubleshooting hosted clusters on OpenShift Virtualization {#hcp-ts-ocp-virt_hcp-troubleshooting} When you troubleshoot a hosted cluster on OpenShift Virtualization, start with the top-level `HostedCluster` and `NodePool` resources and then work down the stack until you find the root cause. The following steps can help you discover the root cause of common issues. ### Troubleshooting HostedCluster resource stuck in a partial state {#hcp-ts-hc-stuck_hcp-troubleshooting} If a hosted control plane is not coming fully online because a `HostedCluster` resource is pending, identify the problem by checking prerequisites, resource conditions, and node and Operator status. **Procedure** - Ensure that you meet all of the prerequisites for a hosted cluster on OpenShift Virtualization. - View the conditions on the `HostedCluster` and `NodePool` resources for validation errors that prevent progress. - By using the `kubeconfig` file of the hosted cluster, inspect the status of the hosted cluster: - View the output of the `oc get clusteroperators` command to see which cluster Operators are pending. - View the output of the `oc get nodes` command to ensure that worker nodes are ready. ### Identifying why no compute nodes are registered {#hcp-ts-no-nodes-reg_hcp-troubleshooting} If a hosted control plane is not coming fully online because the hosted control plane has no compute nodes registered, identify the problem by checking the status of various parts of the hosted control plane. **Procedure** - View the `HostedCluster` and `NodePool` conditions for failures that indicate what the problem might be. - Enter the following command to view the KubeVirt compute node virtual machine (VM) status for the `NodePool` resource: ```terminal $ oc get vm -n ``` - If the VMs are stuck in the provisioning state, enter the following command to view the CDI import pods within the VM namespace for clues about why the importer pods have not completed: ```terminal $ oc get pods -n | grep "import" ``` - If the VMs are stuck in the starting state, enter the following command to view the status of the virt-launcher pods: ```terminal $ oc get pods -n -l kubevirt.io=virt-launcher ``` If the virt-launcher pods are in a pending state, investigate why the pods are not being scheduled. For example, not enough resources might exist to run the virt-launcher pods. - If the VMs are running but they are not registered as compute nodes, use the web console to gain VNC access to one of the affected VMs. The VNC output indicates whether the ignition configuration was applied. If a VM cannot access the hosted control plane ignition server on startup, the VM cannot be provisioned correctly. - If the ignition configuration was applied but the VM is still not registering as a node, see "Identifying the problem: Access the VM console logs" to learn how to access the VM console logs during startup. **Additional resources** {._additional-resources} - [Identifying the problem: Access the VM console logs](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.11/html/clusters/cluster_mce_overview#identifying-vm-console-logs) ### Identifying why compute nodes are not ready {#hcp-ts-nodes-stuck_hcp-troubleshooting} During cluster creation, nodes enter the `NotReady` state temporarily while the networking stack is rolled out. This part of the process is normal. However, if this part of the process takes longer than 15 minutes, identify the problem by investigating the node object and pods. **Procedure** 1. Enter the following command to view the conditions on the node object and determine why the node is not ready: ```terminal $ oc get nodes -o yaml ``` 2. Enter the following command to look for failing pods within the cluster: ```terminal $ oc get pods -A --field-selector=status.phase!=Running,status,phase!=Succeeded ``` ### Identifying why ingress and console cluster Operators are not coming online {#hcp-ts-ingress-not-online_hcp-troubleshooting} If a hosted control plane is not coming fully online because the Ingress and console cluster Operators are not online, check the wildcard DNS routes and load balancer. **Procedure** - If the cluster uses the default Ingress behavior, enter the following command to ensure that wildcard DNS routes are enabled on the OpenShift Container Platform cluster that the virtual machines (VMs) are hosted on: ```terminal $ oc patch ingresscontroller -n openshift-ingress-operator \ default --type=json -p \ '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]' ``` - If you use a custom base domain for the hosted control plane, complete the following steps: - Ensure that the load balancer is targeting the VM pods correctly. - Ensure that the wildcard DNS entry is targeting the load balancer IP address. ### Identifying why load balancer services for the hosted cluster are unavailable {#hcp-ts-load-balancer-svcs_hcp-troubleshooting} If a hosted control plane is not coming fully online because the load balancer services are not becoming available, check events, details, and the Kubernetes Cluster Configuration Manager (KCCM) pod. **Procedure** - Look for events and details that are associated with the load balancer service within the hosted cluster. - By default, load balancers for the hosted cluster are handled by the kubevirt-cloud-controller-manager within the hosted control plane namespace. Ensure that the KCCM pod is online and view its logs for errors or warnings. To identify the KCCM pod in the hosted control plane namespace, enter the following command: ```terminal $ oc get pods -n \ -l app=cloud-controller-manager ``` ### Identifying why hosted cluster PVCs are not available {#hcp-ts-pvcs-not-avail_hcp-troubleshooting} If a hosted control plane is not coming fully online because the persistent volume claims (PVCs) for a hosted cluster are not available, check the PVC events and details, and component logs. **Procedure** - Look for events and details that are associated with the PVC to understand which errors are occurring. - If a PVC is failing to attach to a pod, view the logs for the kubevirt-csi-node `daemonset` component within the hosted cluster to further investigate the problem. To identify the kubevirt-csi-node pods for each node, enter the following command: ```terminal $ oc get pods -n openshift-cluster-csi-drivers -o wide \ -l app=kubevirt-csi-driver ``` - If a PVC cannot bind to a persistent volume (PV), view the logs of the kubevirt-csi-controller component within the hosted control plane namespace. To identify the kubevirt-csi-controller pod within the hosted control plane namespace, enter the following command: ```terminal $ oc get pods -n -l app=kubevirt-csi-driver ``` ### Identifying why VM nodes are not joining the cluster {#hcp-ts-vm-nodes_hcp-troubleshooting} If a hosted control plane is not coming fully online because the virtual machine (VM) nodes are not correctly joining the cluster, access the VM console logs. **Procedure** - To access the VM console logs, complete the steps in "How to get serial console logs for VMs part of OpenShift Virtualization Hosted Control Plane clusters". **Additional resources** {._additional-resources} - [How to get serial console logs for VMs part of OpenShift Virtualization Hosted Control Plane clusters (Red Hat Knowledgebase)](https://access.redhat.com/solutions/7037705) ### Resolving RHCOS image mirroring failures {#hcp-ts-rhcos_hcp-troubleshooting} For hosted control planes on OpenShift Virtualization in a disconnected environment, if `oc-mirror` fails to automatically mirror the Red Hat Enterprise Linux CoreOS (RHCOS) image to the internal registry, you can manually mirror the RHCOS image to the internal registry. When you create your first hosted cluster, the Kubevirt virtual machine does not boot, because the boot image is not available in the internal registry. To resolve this issue, manually mirror the RHCOS image to the internal registry. **Procedure** 1. Get the internal registry name by running the following command: ```terminal $ oc get imagecontentsourcepolicy -o json \ | jq -r '.items[].spec.repositoryDigestMirrors[0].mirrors[0]' ``` 2. Get a payload image by running the following command: ```terminal $ oc get clusterversion version -ojsonpath='{.status.desired.image}' ``` 3. Extract the `0000_50_installer_coreos-bootimages.yaml` file that contains boot images from your payload image on the hosted cluster. Replace `` with the name of your payload image. Run the following command: ```terminal $ oc image extract \ --file /release-manifests/0000_50_installer_coreos-bootimages.yaml \ --confirm ``` 4. Get the RHCOS image by running the following command: ```terminal $ cat 0000_50_installer_coreos-bootimages.yaml | yq -r .data.stream \ | jq -r '.architectures.x86_64.images.kubevirt."digest-ref"' ``` 5. Mirror the RHCOS image to your internal registry by running the following command: ```terminal $ oc image mirror ``` - Replace `` with your RHCOS image; for example, `quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9643ead36b1c026be664c9c65c11433c6cdf71bfd93ba229141d134a4a6dd94`. - Replace `` with the name of your internal registry; for example, `virthost.ostest.test.metalkube.org:5000/localimages/ocp-v4.0-art-dev`. 6. Create a YAML file named `rhcos-boot-kubevirt.yaml` that defines the `ImageDigestMirrorSet` object. See the following example configuration: ```yaml apiVersion: config.openshift.io/v1 kind: ImageDigestMirrorSet metadata: name: rhcos-boot-kubevirt spec: repositoryDigestMirrors: - mirrors: - virthost.ostest.test.metalkube.org:5000/localimages/ocp-v4.0-art-dev source: quay.io/openshift-release-dev/ocp-v4.0-art-dev ``` - `spec.repositoryDigestMirrors.mirrors` specifies the name of your internal registry. - `spec.repositoryDigestMirrors.source` specifies your RHCOS image without its digest. 7. Apply the `rhcos-boot-kubevirt.yaml` file to create the `ImageDigestMirrorSet` object by running the following command: ```terminal $ oc apply -f rhcos-boot-kubevirt.yaml ``` ### Returning non-bare-metal clusters to the late binding pool {#hcp-ts-non-bm_hcp-troubleshooting} If you are using late binding managed clusters without `BareMetalHosts`, you must complete additional manual steps to delete a late binding cluster and return the nodes back to the Discovery ISO. For late binding managed clusters without `BareMetalHosts`, removing cluster information does not automatically return all nodes to the Discovery ISO. To unbind the non-bare-metal nodes with late binding, complete the following steps. **Procedure** 1. Remove the cluster information. For more information, see "Removing a cluster from management". 2. Clean the root disks. 3. Reboot manually with the Discovery ISO. **Additional resources** {._additional-resources} - [Removing a cluster from management](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/latest/html/clusters/cluster_mce_overview#remove-managed-cluster) ## Troubleshooting hosted clusters on bare metal {#hcp-ts-bm_hcp-troubleshooting} If you encounter issues with hosted control planes on bare metal, review the troubleshooting procedures to diagnose and resolve them. ### Determining why nodes are not added to a hosted cluster on bare metal {#hcp-ts-bm-nodes-not-added_hcp-troubleshooting} When you scale up a hosted cluster with nodes that were provisioned by using Assisted Installer, the host fails to pull the ignition with a URL that contains port `22642`. That URL is invalid for hosted control planes and indicates that an issue exists with the cluster. **Procedure** 1. To determine the issue, review the assisted-service logs by entering the following command: ```terminal $ oc logs -n multicluster-engine ``` Replace `` with the Assisted Service pod name. 2. In the logs, find errors that resemble these examples: ```terminal error="failed to get pull secret for update: invalid pull secret data in secret pull-secret" ``` ```terminal pull secret must contain auth for \"registry.redhat.io\" ``` 3. To fix this issue, see "Add the pull secret to the namespace" in the multicluster engine for Kubernetes Operator documentation. > [!NOTE] > To use hosted control planes, you must have multicluster engine Operator installed, either as a standalone Operator or as part of Red Hat Advanced Cluster Management. Because the Operator has a close association with Red Hat Advanced Cluster Management, the documentation for the Operator is published within that product’s documentation. Even if you do not use Red Hat Advanced Cluster Management, the parts of its documentation that cover multicluster engine Operator are relevant to hosted control planes. **Additional resources** {._additional-resources} - [Add the pull secret to the namespace](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.16/html-single/clusters/index#on-prem-creating-your-cluster-with-the-cli-pull-secret) ## Restarting hosted control plane components {#hosted-restart-hcp-components_hcp-troubleshooting} If you are an administrator for hosted control planes, you can use the `hypershift.openshift.io/restart-date` annotation to restart all control plane components for a particular `HostedCluster` resource. For example, you might need to restart control plane components for certificate rotation. **Procedure** - To restart a control plane, annotate the `HostedCluster` resource by entering the following command: ```terminal $ oc annotate hostedcluster \ -n \ \ hypershift.openshift.io/restart-date=$(date --iso-8601=seconds) ``` The control plane is restarted whenever the value of the annotation changes. The `date` command serves as the source of a unique string. The annotation is treated as a string, not a timestamp. **Verification** After you restart a control plane, the following hosted control planes components are typically restarted: > [!NOTE] > You might see some additional components restarting as a side effect of changes implemented by the other components. - catalog-operator - certified-operators-catalog - cluster-api - cluster-autoscaler - cluster-policy-controller - cluster-version-operator - community-operators-catalog - control-plane-operator - hosted-cluster-config-operator - ignition-server - ingress-operator - konnectivity-agent - konnectivity-server - kube-apiserver - kube-controller-manager - kube-scheduler - machine-approver - oauth-openshift - olm-operator - openshift-apiserver - openshift-controller-manager - openshift-oauth-apiserver - packageserver - redhat-marketplace-catalog - redhat-operators-catalog ## Pausing the reconciliation of a hosted cluster and hosted control plane {#hosted-control-planes-pause-reconciliation_hcp-troubleshooting} If you are a cluster instance administrator, you can pause the reconciliation of a hosted cluster and hosted control plane. You might want to pause reconciliation when you back up and restore an etcd database or when you need to debug problems with a hosted cluster or hosted control plane. **Procedure** 1. To pause reconciliation for a hosted cluster and hosted control plane, populate the `pausedUntil` field of the `HostedCluster` resource. - To pause the reconciliation until a specific time, enter the following command: ```terminal $ oc patch -n \ hostedclusters/ \ -p '{"spec":{"pausedUntil":""}}' \ --type=merge ``` Replace `` with a timestamp in the RFC339 format; for example, `2024-03-03T03:28:48Z`. The reconciliation is paused until the specified time is passed. - To pause the reconciliation indefinitely, enter the following command: ```terminal $ oc patch -n \ hostedclusters/ \ -p '{"spec":{"pausedUntil":"true"}}' \ --type=merge ``` The reconciliation is paused until you remove the field from the `HostedCluster` resource. When the pause reconciliation field is populated for the `HostedCluster` resource, the field is automatically added to the associated `HostedControlPlane` resource. 2. To remove the `pausedUntil` field, enter the following patch command: ```terminal $ oc patch -n \ hostedclusters/ \ -p '{"spec":{"pausedUntil":null}}' \ --type=merge ``` ## Resolving agent service failures for hosted control planes on IBM Z {#agent-service-failure_hcp-troubleshooting} In some cases, agents might fail to join the cluster after booting the machines with the boot artifacts. You can confirm this issue by checking the `agent.service` logs for the following error: ``` Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists ``` This issue occurs because image signature verification fails when no signature is present. As a workaround, you can disable signature verification by modifying the container policy. **Procedure** 1. Add the `ignitionConfigOverride` field in the `InfraEnv` manifest to override the `/etc/containers/policy.json` file. This disables signature verification for container images. 2. Replace the base64-encoded content in the `ignitionConfigOverride` with the required `/etc/containers/policy.json` configuration according to your image registries. See the following example: ```json {title="Example"} { "default": [ { "type": "insecureAcceptAnything" } ], "transports": { "docker": { "": [ { "type": "insecureAcceptAnything" } ], "REGISTRY2": [ { "type": "insecureAcceptAnything" } ] }, "docker-daemon": { "": [ { "type": "insecureAcceptAnything" } ] } } } ``` ```yaml {title="Example InfraEnv manifest with ignitionConfigOverride"} apiVersion: agent-install.openshift.io/v1beta1 kind: InfraEnv metadata: name: namespace: spec: cpuArchitecture: s390x pullSecretRef: name: pull-secret sshAuthorizedKey: ignitionConfigOverride: '{"ignition":{"version":"3.2.0"},"storage":{"files":[{"path":"/etc/containers/policy.json","mode":420,"overwrite":true,"contents":{"source":"data:text/plain;charset=utf-8;base64,ewogICAgImRlZmF1bHQiOiBbCiAgICAgICAgewogICAgICAgICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgICAgIH0KICAgIF0sCiAgICAidHJhbnNwb3J0cyI6CiAgICAgICAgewogICAgICAgICAgICAiZG9ja2VyLWRhZW1vbiI6CiAgICAgICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAgICAgIiI6IFt7InR5cGUiOiJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIn1dCiAgICAgICAgICAgICAgICB9CiAgICAgICAgfQp9"}}]}}' ``` ## Known limitations for internal subnets for hosted clusters {#hcp-ts-internal-subnets_hcp-troubleshooting} Several known limitations exist for internal subnets on hosted clusters. - IPv6 subnets are not supported. - The hosted control planes command-line interface, `hcp`, might not have native flags for the subnet fields. Manual YAML editing or `oc patch` is required. - Modifying OVN subnets after you create a cluster triggers a rollout of OVN components, which might cause brief network disruptions. - You cannot modify OVN subnet configuration while a cluster update is in progress or scheduled. ### Configuring the ovnKubernetesConfig object fails with an error {#hcp-ts-ovnkubernetes_hcp-troubleshooting} When you try to configure the `ovnKubernetesConfig` object on a hosted cluster by using a different network type, such as `OpenShiftSDN`, an error occurs because hosted control planes works only with the `OVNKubernetes` network type. **Procedure** - Verify the network type of your hosted cluster by entering the following command: ```terminal $ oc get hostedcluster -n \ -o jsonpath='{.spec.networking.networkType}' ``` ### Setting CIDR values in internal subnet fields {#hcp-ts-identical-subnet-fields_hcp-troubleshooting} If the `internalJoinSubnet` field and the `internalTransitSwitchSubnet` field are set to the same classless inter-domain routing (CIDR) values, an error occurs. **Procedure** - Use different subnets for each field, as shown in the following example: ```yaml apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster metadata: # ... spec: #... operatorConfiguration: clusterNetworkOperator: ovnKubernetesConfig: ipv4: internalJoinSubnet: "100.99.0.0/16" internalTransitSwitchSubnet: "100.69.0.0/16" # ... ``` ### Ensuring a valid IPv4 CIDR format {#hcp-ts-subnet-cidr-format_hcp-troubleshooting} If you do not specify subnets in a valid classless inter-domain range (CIDR) format, an error occurs. **Procedure** - Ensure that the CIDR format follows the following format: ```text X.X.X.X/Y ``` where: `X` : is a value from `0` to `255`. The first octet must not be `0`. `Y` : is a value from `0` to `30`. ```text {title="Valid examples"} 100.99.0.0/16 192.168.1.0/24 ``` ```text {title="Invalid examples"} 100.99.0.0 256.1.1.0/16 0.99.0.0/16 ``` ### Avoiding an overlap between OVN subnets and CIDR values {#hcp-ts-cidr-overlap_hcp-troubleshooting} If the configured OVN subnets overlap with the machine classless inter-domain routing (CIDR), service CIDR, cluster network CIDR, or with each other, an error occurs. **Procedure** - Use subnets that do not overlap with any network CIDR. You can use a CIDR calculator to verify that no overlaps exist. ```yaml {title="Example of configuration with no overlaps"} spec: networking: machineCIDR: 10.0.0.0/16 serviceCIDR: 172.30.0.0/16 clusterNetwork: - cidr: 10.128.0.0/14 operatorConfiguration: clusterNetworkOperator: ovnKubernetesConfig: ipv4: internalJoinSubnet: "100.99.0.0/16" internalTransitSwitchSubnet: "100.69.0.0/16" ``` ### Resolving a stuck OVN rollout {#hcp-ts-nodes-not-ready_hcp-troubleshooting} After you change an existing configuration, the OVN component rollout might take a long time or encounter issues. **Procedure** 1. Check the status of the `ovnkube-node` DaemonSet rollout by entering the following command: ```terminal $ oc rollout status daemonset/ovnkube-node \ -n openshift-ovn-kubernetes \ --kubeconfig=hosted-kubeconfig ``` 2. Check the pod logs for errors by entering the following command: ```terminal $ oc logs -n openshift-ovn-kubernetes \ -l app=ovnkube-node \ --kubeconfig=hosted-kubeconfig ``` If the rollout is stuck, you might need to revert the configuration change. ## Troubleshooting connectivity for hosted control planes {#hcp-ts-connectivity_hcp-troubleshooting} By using connectivity metrics, you can diagnose whether any issues are related to connectivity from a hosted control plane to a data plane. ### Troubleshooting connectivity from the control plane to the data plane {#hcp-ts-connect-data-plane_hcp-troubleshooting} To diagnose connectivity issues from a hosted control plane to the compute nodes in a data plane, check the status of the `DataPlaneConnectionAvailable` condition. If the status of the `DataPlaneConnectionAvailable` condition is `True`, the control plane can successfully reach the data plane nodes through the `konnectivity-agent` pods. If the status is `False`, take the following steps to determine why the control plane cannot reach the data plane. **Procedure** 1. Check the network policies that might block the `Konnectivity` service traffic. 2. Review the firewall rules between the control plane and the data plane. 3. View the status of the `konnectivity-agent` pods in the data plane. 4. In the control plane, review the `Konnectivity` server deployment. **Additional resources** {._additional-resources} - [Connectivity monitoring for hosted control planes](/openshift-docs-markdown/hosted_control_planes/hcp-observability#hcp-connectivity-metrics_hcp-observability)