--- title: Troubleshooting the installation --- # Troubleshooting the installation {#ipi-install-troubleshooting} Troubleshoot the OpenShift Container Platform installation process to identify and resolve deployment failures. ## Troubleshooting the installation program workflow {#ipi-install-troubleshooting-install-workflow_ipi-install-troubleshooting} Before troubleshooting the installation environment, it is critical to understand the overall flow of the installer-provisioned installation on bare metal. The following diagrams illustrate a troubleshooting flow with a step-by-step breakdown for the environment. **Figure 1. Troubleshooting workflow for `install-config.yaml` file errors or inaccessible RHCOS images** ![Troubleshooting workflow when the \`install-config.yaml\` file has errors or the Red Hat Enterprise Linux CoreOS (RHCOS) images are inaccessible](/openshift-docs-markdown/images/flow1.png) This diagram illustrates a troubleshooting workflow when the `install-config.yaml` file has errors or the Red Hat Enterprise Linux CoreOS (RHCOS) images are inaccessible. See [Troubleshooting `install-config.yaml`](#ipi-install-troubleshooting-install-config_ipi-install-troubleshooting) for troubleshooting suggestions. **Figure 2. Troubleshooting workflow for bootstrap VM issues, bootstrap VMs that cannot boot up the cluster nodes, and inspecting logs** ![Troubleshooting workflow for bootstrap VM issues](/openshift-docs-markdown/images/flow2.png) This diagram illustrates a troubleshooting workflow for [ bootstrap VM issues](#ipi-install-troubleshooting-bootstrap-vm_ipi-install-troubleshooting), [ bootstrap VMs that cannot boot up the cluster nodes](#ipi-install-troubleshooting-bootstrap-vm-cannot-boot_ipi-install-troubleshooting), and [ inspecting logs](#ipi-install-troubleshooting-bootstrap-vm-inspecting-logs_ipi-install-troubleshooting). When installing an OpenShift Container Platform cluster without the `provisioning` network, this workflow does not apply. **Figure 3. Troubleshooting workflow for cluster nodes that will not PXE boot** ![Troubleshooting workflow for cluster nodes that will not PXE boot](/openshift-docs-markdown/images/flow3.png) This diagram illustrates a troubleshooting workflow for [cluster nodes that will not PXE boot](#ipi-install-troubleshooting-cluster-nodes-will-not-pxe_ipi-install-troubleshooting). If installing using Redfish virtual media, each node must meet minimum firmware requirements for the installation program to deploy the node. For additional details, see [Firmware requirements for installing with virtual media](/openshift-docs-markdown/installing/installing_bare_metal/ipi/ipi-install-prerequisites#ipi-install-firmware-requirements-for-installing-with-virtual-media_ipi-install-prerequisites). **Figure 4. Troubleshooting workflow from a non-accessible API to a validated installation** ![Troubleshooting workflow from a non-accessible API to a validated installation](/openshift-docs-markdown/images/flow4.png) This diagram illustrates a troubleshooting workflow from [ a non-accessible API](#investigating-an-unavailable-kubernetes-api_ipi-install-troubleshooting) to a [validated installation](#ipi-install-troubleshooting-reviewing-the-installation_ipi-install-troubleshooting). ## Troubleshooting `install-config.yaml` {#ipi-install-troubleshooting-install-config_ipi-install-troubleshooting} The `install-config.yaml` configuration file represents all of the nodes that are part of the OpenShift Container Platform cluster. The file contains the necessary options consisting of but not limited to `apiVersion`, `baseDomain`, `imageContentSources` and virtual IP addresses. If errors occur early in the deployment of the OpenShift Container Platform cluster, the errors are likely in the `install-config.yaml` configuration file. **Procedure** 1. Use the guidelines in [YAML-tips](https://www.redhat.com/sysadmin/yaml-tips). 2. Verify the YAML syntax is correct using [syntax-check](http://www.yamllint.com/). 3. Verify the Red Hat Enterprise Linux CoreOS (RHCOS) QEMU images are properly defined and accessible via the URL provided in the `install-config.yaml` by running the following command: ```terminal $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu..qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7 ``` If the output is `200`, there is a valid response from the webserver storing the bootstrap VM image. ## Troubleshooting bootstrap VM issues {#ipi-install-troubleshooting-bootstrap-vm_ipi-install-troubleshooting} The OpenShift Container Platform installation program spawns a bootstrap node virtual machine, which handles provisioning the OpenShift Container Platform cluster nodes. **Procedure** 1. About 10 to 15 minutes after triggering the installation program, ensure the bootstrap VM is operational by running the `virsh` command: ```terminal $ sudo virsh list ``` ```terminal {title="Example output"} Id Name State -------------------------------------------- 12 openshift-xf6fq-bootstrap running ``` > [!NOTE] > The name of the bootstrap VM is always the cluster name followed by a random set of characters and ending in the word "bootstrap." 2. If the bootstrap VM is not running after 10-15 minutes, verify `libvirtd` is running on the system by using the following command: ```terminal $ systemctl status libvirtd ``` ```terminal {title="Example output"} ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago Docs: man:libvirtd(8) https://libvirt.org Main PID: 9850 (libvirtd) Tasks: 20 (limit: 32768) Memory: 74.8M CGroup: /system.slice/libvirtd.service ├─ 9850 /usr/sbin/libvirtd ``` If the bootstrap VM is operational, log in to it. 3. Find the IP address of the bootstrap VM by running the following command: ```terminal $ sudo virsh console example.com ``` ```terminal {title="Example output"} Connected to domain example.com Escape character is ^] Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3 SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519) SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA) SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA) ens3: fd35:919d:4042:2:c7ed:9a9f:a9ec:7 ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f localhost login: ``` > [!IMPORTANT] > When deploying an OpenShift Container Platform cluster without the `provisioning` network, you must use a public IP address and not a private IP address like `172.22.0.2`. 4. After you obtain the IP address, log in to the bootstrap VM by running the `ssh` command: > [!NOTE] > In the console output of the previous step, you can use the IPv6 IP address provided by `ens3` or the IPv4 IP provided by `ens4`. ```terminal $ ssh core@172.22.0.2 ``` If you are not successful logging in to the bootstrap VM, you have likely encountered one of the following scenarios: - You cannot reach the `172.22.0.0/24` network. Verify the network connectivity between the provisioner and the `provisioning` network bridge. This issue might occur if you are using a `provisioning` network. - You cannot reach the bootstrap VM through the public network. When attempting to SSH via `baremetal` network, verify connectivity on the `provisioner` host specifically around the `baremetal` network bridge. - You encountered `Permission denied (publickey,password,keyboard-interactive)`. When attempting to access the bootstrap VM, a `Permission denied` error might occur. Verify that the SSH key for the user attempting to log in to the VM is set within the `install-config.yaml` file. ### Bootstrap VM cannot boot up the cluster nodes {#ipi-install-troubleshooting-bootstrap-vm-cannot-boot_ipi-install-troubleshooting} During the deployment, it is possible for the bootstrap VM to fail to boot the cluster nodes, which prevents the VM from provisioning the nodes with the RHCOS image. This scenario can arise due to: - A problem with the `install-config.yaml` file. - Issues with out-of-band network access when using the baremetal network. To verify the issue, there are three containers related to `ironic`: - `ironic` - `ironic-inspector` **Procedure** 1. Log in to the bootstrap VM by running the following command: ```terminal $ ssh core@172.22.0.2 ``` 2. Check the container logs by running the following command: ```terminal [core@localhost ~]$ sudo podman logs -f ``` Replace `` with one of `ironic` or `ironic-inspector`. If you encounter an issue where the control plane nodes are not booting up from PXE, check the `ironic` pod. The `ironic` pod contains information about the attempt to boot the cluster nodes, because it attempts to log in to the node over IPMI. **Results** The cluster nodes might be in the `ON` state when deployment started. **Troubleshooting** Power off the OpenShift Container Platform cluster nodes before you begin the installation over IPMI by running the following command: ```terminal $ ipmitool -I lanplus -U root -P -H power off ``` ### Inspecting logs {#ipi-install-troubleshooting-bootstrap-vm-inspecting-logs_ipi-install-troubleshooting} When experiencing issues downloading or accessing the RHCOS images, first verify that the URL is correct in the `install-config.yaml` configuration file. See the following example of internal webserver hosting RHCOS images: ```yaml bootstrapOSImage: http:///rhcos-43.81.202001142154.0-qemu..qcow2.gz?sha256=9d999f55ff1d44f7ed7c106508e5deecd04dc3c06095d34d36bf1cd127837e0c clusterOSImage: http:///rhcos-43.81.202001142154.0-openstack..qcow2.gz?sha256=a1bda656fa0892f7b936fdc6b6a6086bddaed5dafacedcd7a1e811abb78fe3b0 ``` The `coreos-downloader` container downloads resources from a webserver or from the external [quay.io](https://quay.io) registry, whichever the `install-config.yaml` configuration file specifies. Verify that the `coreos-downloader` container is up and running and inspect its logs as needed. **Procedure** 1. Log in to the bootstrap VM by running the following command: ```terminal $ ssh core@172.22.0.2 ``` 2. Check the status of the `coreos-downloader` container within the bootstrap VM by running the following command: ```terminal [core@localhost ~]$ sudo podman logs -f coreos-downloader ``` If the bootstrap VM cannot access the URL to the images, use the `curl` command to verify that the VM can access the images. 3. Inspect the system and bootstrap logs to verify if all the containers launched during the deployment phase: 1. Inspect the system logs by running the following command: ```terminal [core@localhost ~]$ journalctl -xe ``` 2. Inspect the `bootkube` logs to check bootstrap progress by running the following command: ```terminal [core@localhost ~]$ journalctl -b -f -u bootkube.service ``` 4. Verify all the pods, including `dnsmasq`, `mariadb`, `httpd`, and `ironic`, are running by using the following command: ```terminal [core@localhost ~]$ sudo podman ps ``` 5. If there are issues with the pods, check the logs of the containers with issues. To check the logs of the `ironic` service, run the following command: ```terminal [core@localhost ~]$ sudo podman logs ironic ``` ## Investigating an unavailable Kubernetes API {#investigating-an-unavailable-kubernetes-api_ipi-install-troubleshooting} When the Kubernetes API is unavailable, check the control plane nodes to ensure that they are running the correct components. Also, check the hostname resolution. **Procedure** 1. Ensure that `etcd` is running on each of the control plane nodes by running the following command: ```terminal $ sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet) ``` 2. If the previous command fails, ensure that Kubelet created the `etcd` pods by running the following command: ```terminal $ sudo crictl pods --name=etcd-member ``` If there are no pods, investigate `etcd`. 3. Check the cluster nodes to ensure they have a fully qualified domain name, and not just `localhost.localdomain`, by using the following command: ```terminal $ hostname ``` If a hostname is not set, set the correct hostname by running the following command: ```terminal $ sudo hostnamectl set-hostname ``` 4. Ensure that each node has the correct name resolution in the DNS server by running the `dig` command: ```terminal $ dig api..example.com ``` ```terminal {title="Example output"} ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api..example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good) ;; QUESTION SECTION: ;api..example.com. IN A ;; ANSWER SECTION: api..example.com. 10800 IN A 10.19.13.86 ;; AUTHORITY SECTION: .example.com. 10800 IN NS .example.com. ;; ADDITIONAL SECTION: .example.com. 10800 IN A 10.19.14.247 ;; Query time: 0 msec ;; SERVER: 10.19.14.247#53(10.19.14.247) ;; WHEN: Tue May 19 20:30:59 UTC 2020 ;; MSG SIZE rcvd: 140 ``` The output in the foregoing example indicates that the appropriate IP address for the `api..example.com` VIP is `10.19.13.86`. This IP address should reside on the `baremetal` network. ## Troubleshooting a failure to initialize the cluster {#troubleshooting-failure-to-initialize-the-cluster_ipi-install-troubleshooting} The installation program uses the Cluster Version Operator to create all the components of an OpenShift Container Platform cluster. When the installation program fails to initialize the cluster, you can retrieve the most important information from the `ClusterVersion` and `ClusterOperator` objects. **Procedure** 1. Inspect the `ClusterVersion` object by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion -o yaml ``` ```yaml {title="Example output"} apiVersion: config.openshift.io/v1 kind: ClusterVersion metadata: creationTimestamp: 2019-02-27T22:24:21Z generation: 1 name: version resourceVersion: "19927" selfLink: /apis/config.openshift.io/v1/clusterversions/version uid: 6e0f4cf8-3ade-11e9-9034-0a923b47ded4 spec: channel: stable-4.1 clusterID: 5ec312f9-f729-429d-a454-61d4906896ca status: availableUpdates: null conditions: - lastTransitionTime: 2019-02-27T22:50:30Z message: Done applying 4.1.1 status: "True" type: Available - lastTransitionTime: 2019-02-27T22:50:30Z status: "False" type: Failing - lastTransitionTime: 2019-02-27T22:50:30Z message: Cluster version is 4.1.1 status: "False" type: Progressing - lastTransitionTime: 2019-02-27T22:24:31Z message: 'Unable to retrieve available updates: unknown version 4.1.1 reason: RemoteFailed status: "False" type: RetrievedUpdates desired: image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a version: 4.1.1 history: - completionTime: 2019-02-27T22:50:30Z image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a startedTime: 2019-02-27T22:24:31Z state: Completed version: 4.1.1 observedGeneration: 1 versionHash: Wa7as_ik1qE= ``` 2. View the conditions by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion version \ -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}' ``` Some of most important conditions include `Failing`, `Available` and `Progressing`. ```terminal {title="Example output"} Available True Done applying 4.1.1 Failing False Progressing False Cluster version is 4.0.0-0.alpha-2019-02-26-194020 RetrievedUpdates False Unable to retrieve available updates: unknown version 4.1.1 ``` 3. Inspect the `ClusterOperator` object by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator ``` The command returns the status of the cluster Operators. ```terminal {title="Example output"} NAME VERSION AVAILABLE PROGRESSING FAILING SINCE cluster-baremetal-operator True False False 17m cluster-autoscaler True False False 17m cluster-storage-operator True False False 10m console True False False 7m21s dns True False False 31m image-registry True False False 9m58s ingress True False False 10m kube-apiserver True False False 28m kube-controller-manager True False False 21m kube-scheduler True False False 25m machine-api True False False 17m machine-config True False False 17m marketplace-operator True False False 10m monitoring True False False 8m23s network True False False 13m node-tuning True False False 11m openshift-apiserver True False False 15m openshift-authentication True False False 20m openshift-cloud-credential-operator True False False 18m openshift-controller-manager True False False 10m openshift-samples True False False 8m42s operator-lifecycle-manager True False False 17m service-ca True False False 30m ``` 4. Inspect individual cluster Operators by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator -oyaml ``` Replace `` with the name of a cluster Operator. This command is useful for identifying why a cluster Operator has not achieved the `Available` state or is in the `Failed` state. ```yaml {title="Example output"} apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: creationTimestamp: 2019-02-27T22:47:04Z generation: 1 name: monitoring resourceVersion: "24677" selfLink: /apis/config.openshift.io/v1/clusteroperators/monitoring uid: 9a6a5ef9-3ae1-11e9-bad4-0a97b6ba9358 spec: {} status: conditions: - lastTransitionTime: 2019-02-27T22:49:10Z message: Successfully rolled out the stack. status: "True" type: Available - lastTransitionTime: 2019-02-27T22:49:10Z status: "False" type: Progressing - lastTransitionTime: 2019-02-27T22:49:10Z status: "False" type: Failing extension: null relatedObjects: null version: "" ``` 5. Get the cluster Operator’s status condition by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator \ -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}' ``` Replace `` with the name of one of the operators above. ```terminal {title="Example output"} Available True Successfully rolled out the stack Progressing False Failing False ``` 6. Retrieve the list of objects owned by the cluster Operator by running the following command: ```terminal oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator kube-apiserver \ -o=jsonpath='{.status.relatedObjects}' ``` ```javascript {title="Example output"} [map[resource:kubeapiservers group:operator.openshift.io name:cluster] map[group: name:openshift-config resource:namespaces] map[group: name:openshift-config-managed resource:namespaces] map[group: name:openshift-kube-apiserver-operator resource:namespaces] map[group: name:openshift-kube-apiserver resource:namespaces]] ``` ## Troubleshooting a failure to fetch the console URL {#troubleshooting-failure-to-fetch-the-console-url_ipi-install-troubleshooting} The installation program retrieves the URL for the OpenShift Container Platform console by using `[route][route-object]` within the `openshift-console` namespace. If the installation program fails the retrieve the URL for the console, use the following procedure. **Procedure** 1. Check if the console router is in the `Available` or `Failing` state by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator console -oyaml ``` ```yaml {title="Example output"} apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: creationTimestamp: 2019-02-27T22:46:57Z generation: 1 name: console resourceVersion: "19682" selfLink: /apis/config.openshift.io/v1/clusteroperators/console uid: 960364aa-3ae1-11e9-bad4-0a97b6ba9358 spec: {} status: conditions: - lastTransitionTime: 2019-02-27T22:46:58Z status: "False" type: Failing - lastTransitionTime: 2019-02-27T22:50:12Z status: "False" type: Progressing - lastTransitionTime: 2019-02-27T22:50:12Z status: "True" type: Available - lastTransitionTime: 2019-02-27T22:46:57Z status: "True" type: Upgradeable extension: null relatedObjects: - group: operator.openshift.io name: cluster resource: consoles - group: config.openshift.io name: cluster resource: consoles - group: oauth.openshift.io name: console resource: oauthclients - group: "" name: openshift-console-operator resource: namespaces - group: "" name: openshift-console resource: namespaces versions: null ``` 2. Manually retrieve the console URL by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get route console -n openshift-console \ -o=jsonpath='{.spec.host}' console-openshift-console.apps.adahiya-1.devcluster.openshift.com ``` ## Troubleshooting a failure to add the ingress certificate to kubeconfig {#troubleshooting-failure-to-add-the-ingress-certificate-to-kubeconfig_ipi-install-troubleshooting} The installation program adds the default ingress certificate to the list of trusted client certificate authorities in `${INSTALL_DIR}/auth/kubeconfig`. If the installation program fails to add the ingress certificate to the `kubeconfig` file, you can retrieve the certificate from the cluster and add it. **Procedure** 1. Retrieve the certificate from the cluster by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get configmaps default-ingress-cert \ -n openshift-config-managed -o=jsonpath='{.data.ca-bundle\.crt}' ``` ```terminal {title="Example output"} -----BEGIN CERTIFICATE----- MIIC/TCCAeWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMSwwKgYDVQQDDCNjbHVz dGVyLWluZ3Jlc3Mtb3BlcmF0b3JAMTU1MTMwNzU4OTAeFw0xOTAyMjcyMjQ2Mjha Fw0yMTAyMjYyMjQ2MjlaMC4xLDAqBgNVBAMMI2NsdXN0ZXItaW5ncmVzcy1vcGVy YXRvckAxNTUxMzA3NTg5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA uCA4fQ+2YXoXSUL4h/mcvJfrgpBfKBW5hfB8NcgXeCYiQPnCKblH1sEQnI3VC5Pk 2OfNCF3PUlfm4i8CHC95a7nCkRjmJNg1gVrWCvS/ohLgnO0BvszSiRLxIpuo3C4S EVqqvxValHcbdAXWgZLQoYZXV7RMz8yZjl5CfhDaaItyBFj3GtIJkXgUwp/5sUfI LDXW8MM6AXfuG+kweLdLCMm3g8WLLfLBLvVBKB+4IhIH7ll0buOz04RKhnYN+Ebw tcvFi55vwuUCWMnGhWHGEQ8sWm/wLnNlOwsUz7S1/sW8nj87GFHzgkaVM9EOnoNI gKhMBK9ItNzjrP6dgiKBCQIDAQABoyYwJDAOBgNVHQ8BAf8EBAMCAqQwEgYDVR0T AQH/BAgwBgEB/wIBADANBgkqhkiG9w0BAQsFAAOCAQEAq+vi0sFKudaZ9aUQMMha CeWx9CZvZBblnAWT/61UdpZKpFi4eJ2d33lGcfKwHOi2NP/iSKQBebfG0iNLVVPz vwLbSG1i9R9GLdAbnHpPT9UG6fLaDIoKpnKiBfGENfxeiq5vTln2bAgivxrVlyiq +MdDXFAWb6V4u2xh6RChI7akNsS3oU9PZ9YOs5e8vJp2YAEphht05X0swA+X8V8T C278FFifpo0h3Q0Dbv8Rfn4UpBEtN4KkLeS+JeT+0o2XOsFZp7Uhr9yFIodRsnNo H/Uwmab28ocNrGNiEVaVH6eTTQeeZuOdoQzUbClElpVmkrNGY0M42K0PvOQ/e7+y AQ== -----END CERTIFICATE----- ``` 2. Add the certificate to the `client-certificate-authority-data` field in the `${INSTALL_DIR}/auth/kubeconfig` file. ## Troubleshooting SSH access to cluster nodes {#troubleshooting-ssh-access-to-cluster-nodes_ipi-install-troubleshooting} For added security, you cannot SSH into the cluster from outside the cluster by default. However, you can access control plane and worker nodes from the provisioner node. If you cannot SSH into the cluster nodes from the provisioner node, the nodes might be waiting on the bootstrap VM. The control plane nodes retrieve their boot configuration from the bootstrap VM, and they cannot boot successfully if they do not retrieve the boot configuration. **Procedure** 1. If you have physical access to the nodes, check their console output to verify if they have successfully booted. If the nodes are still retrieving their boot configuration, there might be problems with the bootstrap VM . 2. Ensure you have configured the `sshKey: ''` setting in the `install-config.yaml` file, where `` is the public key of the `kni` user on the provisioner node. ## Cluster nodes do not PXE boot {#ipi-install-troubleshooting-cluster-nodes-will-not-pxe_ipi-install-troubleshooting} When OpenShift Container Platform cluster nodes do not PXE boot, run the troubleshooting checks on the cluster nodes that do not PXE boot. This procedure does not apply when installing an OpenShift Container Platform cluster without the `provisioning` network. **Procedure** 1. Check the network connectivity to the `provisioning` network. 2. Ensure PXE is enabled on the NIC for the `provisioning` network and PXE is disabled for all other NICs. 3. Verify that the `install-config.yaml` configuration file includes the `rootDeviceHints` parameter and boot MAC address for the NIC connected to the `provisioning` network. For example: - Control plane node settings ```terminal bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC ``` - Compute node settings ```terminal bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC ``` ## Installing creates no worker nodes {#installing-creates-no-worker-nodes_ipi-install-troubleshooting} The installation program does not provision worker nodes directly. Instead, the Machine API Operator scales nodes up and down on supported platforms. If worker nodes are not created after 15 to 20 minutes, depending on the speed of the cluster’s internet connection, investigate the Machine API Operator. **Procedure** 1. Check the Machine API Operator by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \ --namespace=openshift-machine-api get deployments ``` If `${INSTALL_DIR}` is not set in your environment, replace the value with the name of the installation directory. ```terminal {title="Example output"} NAME READY UP-TO-DATE AVAILABLE AGE cluster-autoscaler-operator 1/1 1 1 86m cluster-baremetal-operator 1/1 1 1 86m machine-api-controllers 1/1 1 1 85m machine-api-operator 1/1 1 1 86m ``` 2. Check the machine controller logs by running the following command: ```terminal $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \ --namespace=openshift-machine-api logs deployments/machine-api-controllers \ --container=machine-controller ``` ## Troubleshooting the Cluster Network Operator {#troubleshooting-the-cluster-network-operator_ipi-install-troubleshooting} The Cluster Network Operator is responsible for deploying the networking components. It runs early in the installation process, after the control plane nodes have come up but before the installation program removes the bootstrap control plane. Issues with this Operator might indicate installation program issues. **Procedure** 1. Ensure the network configuration exists by running the following command: ```terminal $ oc get network -o yaml cluster ``` If it does not exist, the installation program did not create it. To find the reason, run the following command: ```terminal $ openshift-install create manifests ``` Review the manifests to determine why the installation program did not create the network configuration. 2. Ensure the network is running by entering the following command: ```terminal $ oc get po -n openshift-network-operator ``` ## Unable to discover new bare-metal hosts using the BMC {#unable-to-discover-new-bare-metal-hosts-using-the-bmc_ipi-install-troubleshooting} In some cases, the installation program is not be able to discover the new bare-metal hosts and issue an error, because it cannot mount the remote virtual media share. For example: ```terminal ProvisioningError 51s metal3-baremetal-controller Image provisioning failed: Deploy step deploy.deploy failed with BadRequestError: HTTP POST https:///redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia returned code 400. Base.1.8.GeneralError: A general error has occurred. See ExtendedInfo for more information Extended information: [ { "Message": "Unable to mount remote share https:///redfish/boot-.iso.", "MessageArgs": [ "https:///redfish/boot-.iso" ], "MessageArgs@odata.count": 1, "MessageId": "IDRAC.2.5.RAC0720", "RelatedProperties": [ "#/Image" ], "RelatedProperties@odata.count": 1, "Resolution": "Retry the operation.", "Severity": "Informational" } ]. ``` In this situation, if you are using virtual media with an unknown certificate authority, you can configure your baseboard management controller (BMC) remote file share settings to trust an unknown certificate authority to avoid this error. > [!NOTE] > This resolution was tested on OpenShift Container Platform 4.11 with Dell iDRAC 9 and firmware version 5.10.50. ## Troubleshooting worker nodes that cannot join the cluster {#worker-nodes-cannot-join-the-cluster_ipi-install-troubleshooting} Installer-provisioned clusters deploy with a DNS server that includes a DNS entry for the `api-int..` URL. If the nodes within the cluster use an external or upstream DNS server to resolve the `api-int..` URL and there is no such entry, worker nodes might fail to join the cluster. Ensure that all nodes in the cluster can resolve the domain name. **Procedure** 1. Add a DNS A/AAAA or CNAME record to internally identify the API load balancer. For example, when using dnsmasq, modify the `dnsmasq.conf` configuration file by running the following command: ```terminal $ sudo nano /etc/dnsmasq.conf ``` ```terminal {title="Example output"} address=/api-int../ address=/api-int.mycluster.example.com/192.168.1.10 address=/api-int.mycluster.example.com/2001:0db8:85a3:0000:0000:8a2e:0370:7334 ``` 2. Add a DNS PTR record to internally identify the API load balancer. For example, when using dnsmasq, modify the `dnsmasq.conf` configuration file by running the following command: ```terminal $ sudo nano /etc/dnsmasq.conf ``` ```terminal {title="Example output"} ptr-record=.in-addr.arpa,api-int.. ptr-record=10.1.168.192.in-addr.arpa,api-int.mycluster.example.com ``` 3. Restart the DNS server. For example, when using dnsmasq, run the following command: ```terminal $ sudo systemctl restart dnsmasq ``` These records must be resolvable from all the nodes within the cluster. ## Cleaning up previous installations {#ipi-install-troubleshooting-cleaning-up-previous-installations_ipi-install-troubleshooting} You must remove the artifacts from any failed deployment attempt before trying to deploy OpenShift Container Platform again. **Procedure** 1. Power off all bare-metal nodes before installing the OpenShift Container Platform cluster by running the following command: ```terminal $ ipmitool -I lanplus -U -P -H power off ``` 2. Remove all old bootstrap resources if any remain from an earlier deployment attempt by running the following script: ```bash for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'}); do sudo virsh destroy $i; sudo virsh undefine $i; sudo virsh vol-delete $i --pool $i; sudo virsh vol-delete $i.ign --pool $i; sudo virsh pool-destroy $i; sudo virsh pool-undefine $i; done ``` 3. Delete the artifacts that the earlier installation generated by running the following command: ```terminal $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \ .openshift_install.log .openshift_install_state.json ``` 4. Re-create the OpenShift Container Platform manifests by running the following command: ```terminal $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests ``` ## Issues with creating the registry {#ipi-install-troubleshooting-registry-issues_ipi-install-troubleshooting} When creating a disconnected registry, you might encounter a "User Not Authorized" error when attempting to mirror the registry. This error might occur if you fail to append the new authentication to the existing `pull-secret.txt` file. **Procedure** 1. Check to ensure authentication is successful by running the following command: ```terminal $ /usr/local/bin/oc adm release mirror \ -a pull-secret-update.json --from=$UPSTREAM_REPO \ --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \ --to=$LOCAL_REG/$LOCAL_REPO ``` > [!NOTE] > Example output of the variables used to mirror the install images: > > ```terminal > UPSTREAM_REPO=${RELEASE_IMAGE} > LOCAL_REG=: > LOCAL_REPO='ocp4/openshift4' > ``` > > The values of `RELEASE_IMAGE` and `VERSION` were set during the **Retrieving OpenShift Installer** step of the **Setting up the environment for an OpenShift installation** section. 2. After mirroring the registry, confirm that you can access it in your disconnected environment by running the following command: ```terminal $ curl -k -u : https://registry.example.com:/v2/_catalog {"repositories":[""]} ``` ## Miscellaneous issues {#ipi-install-troubleshooting-misc-issues_ipi-install-troubleshooting} Review the miscellaneous OpenShift Container Platform issues to identify and resolve uncommon deployment errors in your cluster. ### Addressing the `runtime network not ready` error {#ipi-install-troubleshooting-runtime-network-not-ready_ipi-install-troubleshooting} Troubleshoot the runtime network not ready error in your OpenShift Container Platform cluster to restore node communication. After the deployment of a cluster you might receive the following error: ```text runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network ``` The Cluster Network Operator is responsible for deploying the networking components in response to a special object created by the installation program. It runs very early in the installation process, after the control plane nodes have come up, but before the bootstrap control plane has been torn down. It can be indicative of more subtle installation program issues, such as long delays in bringing up control plane nodes or issues with `apiserver` communication. **Procedure** 1. Inspect the pods in the `openshift-network-operator` namespace by running the following command: ```terminal $ oc get all -n openshift-network-operator ``` ```terminal {title="Example output"} NAME READY STATUS RESTARTS AGE pod/network-operator-69dfd7b577-bg89v 0/1 ContainerCreating 0 149m ``` 2. On the `provisioner` node, determine that the network configuration exists by running the following command: ```terminal $ kubectl get network.config.openshift.io cluster -oyaml ``` ```yaml {title="Example output"} apiVersion: config.openshift.io/v1 kind: Network metadata: name: cluster spec: serviceNetwork: - 172.30.0.0/16 clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 networkType: OVNKubernetes ``` If it does not exist, the installation program did not create it. To determine why the installation program did not create it, run the following command: ```terminal $ openshift-install create manifests ``` 3. Check that the `network-operator` is running by running the following command: ```terminal $ kubectl -n openshift-network-operator get pods ``` 4. Retrieve the logs by running the following command: ```terminal $ kubectl -n openshift-network-operator logs -l "name=network-operator" ``` On high availability clusters with three or more control plane nodes, the Operator will perform leader election and all other Operators will sleep. For additional details, see [Troubleshooting](https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md). ### Addressing the "No disk found with matching rootDeviceHints" error message {#ipi-install-troubleshooting-no-matching-rootdevicehints_ipi-install-troubleshooting} To ensure the installer can locate and configure the target storage drive on your node, troubleshoot the No disk found with matching `rootDeviceHints` error. After you deploy a cluster, you might receive the following error message: ```text No disk found with matching rootDeviceHints ``` To address the `No disk found with matching rootDeviceHints` error message, run the following temporary workaround: **Procedure** 1. Change the `rootDeviceHints` to `minSizeGigabytes: 300`. 2. After changing the `rootDeviceHints` settings, boot the CoreOS and then verify the disk information by running the following command: ```terminal $ udevadm info /dev/sda ``` If you are using DL360 Gen 10 servers, be aware that they have an SD-card slot that might be assigned the `/dev/sda` device name. If no SD card is present in the server, it can cause conflicts. Ensure that the SD card slot is disabled in the server’s BIOS settings. If the `minSizeGigabytes` workaround is not fulfilling the requirements, you might need to revert `rootDeviceHints` back to `/dev/sda`. This change allows ironic images to boot successfully. An alternative approach to fixing this problem is by using the serial ID of the disk. However, be aware that finding the serial ID can be challenging and might make the configuration file less readable. If you select this path, ensure that you gather the serial ID using the previously documented command and incorporate it into your configuration. ### Addressing the cluster nodes not getting the correct IPv6 address over DHCP error {#ipi-install-troubleshooting-incorrect-ipv6-dhcp_ipi-install-troubleshooting} Troubleshoot and resolve DHCP configuration issues that prevent OpenShift Container Platform cluster nodes from receiving their assigned IPv6 addresses. If the cluster nodes are not getting the correct IPv6 address over DHCP, check the following procedure. **Procedure** 1. Ensure the reserved IPv6 addresses reside outside the DHCP range. 2. In the IP address reservation on the DHCP server, ensure the reservation specifies the correct DHCP Unique Identifier (DUID). For example: ```terminal # This is a dnsmasq dhcp reservation, 'id:00:03:00:01' is the client id and '18:db:f2:8c:d5:9f' is the MAC Address for the NIC id:00:03:00:01:18:db:f2:8c:d5:9f,openshift-master-1,[2620:52:0:1302::6] ``` 3. Ensure that route announcements are working. 4. Ensure that the DHCP server is listening on the required interfaces serving the IP address ranges. ### Addressing the cluster nodes not getting the correct hostname over DHCP error {#ipi-install-troubleshooting-incorrect-hostname-dhcp_ipi-install-troubleshooting} During IPv6 deployment, cluster nodes must get their hostname over DHCP. Sometimes the `NetworkManager` does not assign the hostname immediately. A control plane node might report an error such as: ```text Failed Units: 2 NetworkManager-wait-online.service nodeip-configuration.service ``` This error indicates that the cluster node likely booted without first receiving a hostname from the DHCP server, which causes `kubelet` to boot with a `localhost.localdomain` hostname. To address the error, force the node to renew the hostname. **Procedure** 1. Retrieve the `hostname` by running the following command: ```terminal [core@master-X ~]$ hostname ``` If the hostname is `localhost`, proceed with the following steps: > [!NOTE] > Where `X` is the control plane node number. 2. Force the cluster node to renew the DHCP lease by running the following command: ```terminal [core@master-X ~]$ sudo nmcli con up "" ``` Replace `` with the wired connection corresponding to the `baremetal` network. 3. Check `hostname` again by running the following command: ```terminal [core@master-X ~]$ hostname ``` 4. If the hostname is still `localhost.localdomain`, restart `NetworkManager` by running the following command: ```terminal [core@master-X ~]$ sudo systemctl restart NetworkManager ``` 5. If the hostname is still `localhost.localdomain`, wait a few minutes and check again. If the hostname remains `localhost.localdomain`, repeat the previous steps. 6. Restart the `nodeip-configuration` service by running the following command: ```terminal [core@master-X ~]$ sudo systemctl restart nodeip-configuration.service ``` This service will reconfigure the `kubelet` service with the correct hostname references. 7. Reload the unit files definition since the kubelet changed in the previous step by running the following command: ```terminal [core@master-X ~]$ sudo systemctl daemon-reload ``` 8. Restart the `kubelet` service by running the following command: ```terminal [core@master-X ~]$ sudo systemctl restart kubelet.service ``` 9. Ensure `kubelet` booted with the correct hostname by running the following command: ```terminal [core@master-X ~]$ sudo journalctl -fu kubelet.service ``` If the cluster node is not getting the correct hostname over DHCP after the cluster is up and running, such as during a reboot, the cluster will have a pending `csr`. **Do not** approve a `csr`, or other issues might arise. - **Addressing a `csr`** 1. Get CSRs on the cluster by running the following command: ```terminal $ oc get csr ``` 2. Verify if a pending `csr` contains `Subject Name: localhost.localdomain` by running the following command: ```terminal $ oc get csr -o jsonpath='{.spec.request}' | base64 --decode | openssl req -noout -text ``` 3. Remove any `csr` that contains `Subject Name: localhost.localdomain` by running the following command: ```terminal $ oc delete csr ``` ### Addressing the routes not reaching endpoints error {#ipi-install-troubleshooting-unreachable-routes_ipi-install-troubleshooting} To restore external network connectivity to your applications, troubleshoot OpenShift Container Platform routes that do not reach their service endpoints. During the installation process, it is possible to encounter a Virtual Router Redundancy Protocol (VRRP) conflict. This conflict occurs when a currently active OpenShift Container Platform node from an older deployment uses your current OpenShift Container Platform cluster name without actually being part of the new deployment. For example, a cluster was deployed using the cluster name `openshift`, deploying three control plane nodes and three worker nodes. Later, a separate install uses the same cluster name `openshift`, but this redeployment only installed three control plane nodes, leaving the three worker nodes from a previous deployment in an `ON` state. This might cause a Virtual Router Identifier (VRID) conflict and a VRRP conflict. **Procedure** 1. Get the route by running the following command: ```terminal $ oc get route oauth-openshift ``` 2. Check the service endpoint by running the following command: ```terminal $ oc get svc oauth-openshift ``` ```terminal {title="Example output"} NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE oauth-openshift ClusterIP 172.30.19.162 443/TCP 59m ``` 3. Attempt to reach the service from a control plane node by running the following command: ```terminal [core@master0 ~]$ curl -k https://172.30.19.162 ``` ```terminal {title="Example output"} { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"", "reason": "Forbidden", "details": { }, "code": 403 } ``` 4. Identify the `authentication-operator` errors from the `provisioner` node by running the following command: ```terminal $ oc logs deployment/authentication-operator -n openshift-authentication-operator ``` ```terminal {title="Example output"} Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"225c5bd5-b368-439b-9155-5fd3c0459d98", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting" ``` 5. Ensure that the cluster name for every deployment is unique, ensuring no conflict. 6. Turn off all the rogue nodes which are not part of the cluster deployment that are using the same cluster name. Otherwise, the authentication pod of the OpenShift Container Platform cluster might never start successfully. ### Addressing the failed Ignition during firstboot error {#ipi-install-troubleshooting-failed-ignition-during-firstboot_ipi-install-troubleshooting} Troubleshoot and resolve Ignition configuration failures that occur during the firstboot of your nodes to prevent cluster deployment failures. **Procedure** 1. Connect to the node where the Ignition configuration failed: ```terminal Failed Units: 1 machine-config-daemon-firstboot.service ``` 2. Restart the `machine-config-daemon-firstboot` service by running the following command: ```terminal [core@worker-X ~]$ sudo systemctl restart machine-config-daemon-firstboot.service ``` ### Addressing the NTP out of sync error {#ipi-install-troubleshooting-ntp-out-of-sync_ipi-install-troubleshooting} The deployment of OpenShift Container Platform clusters depends on NTP synchronized clocks among the cluster nodes. Without synchronized clocks, the deployment may fail due to clock drift if the time difference is greater than two seconds. **Procedure** 1. Check for differences in the `AGE` of the cluster nodes by running the following command: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION master-0.cloud.example.com Ready master 145m v1.35.4 master-1.cloud.example.com Ready master 135m v1.35.4 master-2.cloud.example.com Ready master 145m v1.35.4 worker-2.cloud.example.com Ready worker 100m v1.35.4 ``` 2. Check for inconsistent timing delays due to clock drift by running the following command: ```terminal $ oc get bmh -n openshift-machine-api ``` ```terminal {title="Example output showing registration error due to clock drift"} master-1 error registering master-1 ipmi:// ``` 3. Check `System clock synchronized` on a node by running the following command: ```terminal $ sudo timedatectl ``` ```terminal {title="Example output"} Local time: Tue 2020-03-10 18:20:02 UTC Universal time: Tue 2020-03-10 18:20:02 UTC RTC time: Tue 2020-03-10 18:36:53 Time zone: UTC (UTC, +0000) System clock synchronized: no NTP service: active RTC in local TZ: no ``` Ensure that the `System clock synchronized` value is **yes** before proceeding with the installation. 4. Resolve clock drift based on your deployment stage: - **Addressing clock synchronization for clock drift in an existing cluster:** 1. Create a Butane config file including the contents of the `chrony.conf` file to be delivered to the nodes. In the following example, create `99-master-chrony.bu` to add the file to the control plane nodes. You can modify the file for compute nodes or repeat this procedure for the compute role. > [!NOTE] > See "Creating machine configs with Butane" for information about Butane. ```yaml variant: openshift version: 4.22.0 metadata: name: 99-master-chrony labels: machineconfiguration.openshift.io/role: master storage: files: - path: /etc/chrony.conf mode: 0644 overwrite: true contents: inline: | server iburst stratumweight 0 driftfile /var/lib/chrony/drift rtcsync makestep 10 3 bindcmdaddress 127.0.0.1 bindcmdaddress ::1 keyfile /etc/chrony.keys commandkey 1 generatecommandkey noclientlog logchange 0.5 logdir /var/log/chrony ``` Replace `` with the IP address of the NTP server. 2. Use Butane to generate a `MachineConfig` object file, `99-master-chrony.yaml`, containing the configuration to be delivered to the nodes by running the following command: ```terminal $ butane 99-master-chrony.bu -o 99-master-chrony.yaml ``` 3. Apply the `MachineConfig` object file by running the following command: ```terminal $ oc apply -f 99-master-chrony.yaml ``` 4. Ensure the `System clock synchronized` value is **yes** by running the following command: ```terminal $ sudo timedatectl ``` ```terminal {title="Example output"} Local time: Tue 2020-03-10 19:10:02 UTC Universal time: Tue 2020-03-10 19:10:02 UTC RTC time: Tue 2020-03-10 19:36:53 Time zone: UTC (UTC, +0000) System clock synchronized: yes NTP service: active RTC in local TZ: no ``` - **Addressing clock synchronization for clock drift before deployment:** 1. Generate the manifest files and add this file to the `openshift` directory. For example: ```terminal $ cp chrony-masters.yaml ~/clusterconfigs/openshift/99_masters-chrony-configuration.yaml ``` 2. Continue to create the cluster. ## Reviewing the installation {#ipi-install-troubleshooting-reviewing-the-installation_ipi-install-troubleshooting} After installation, ensure the installation program deployed the nodes and pods successfully. **Procedure** 1. When the OpenShift Container Platform cluster nodes are installed appropriately, ensure the following `Ready` state within the `STATUS` column by running the following command: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION master-0.example.com Ready master,worker 4h v1.35.4 master-1.example.com Ready master,worker 4h v1.35.4 master-2.example.com Ready master,worker 4h v1.35.4 ``` 2. Confirm the installation program deployed all pods successfully. Remove any pods that are still running or have completed as part of the output by running the following command: ```terminal $ oc get pods --all-namespaces | grep -iv running | grep -iv complete ```