--- title: Creating a cluster with multi-architecture compute machines on IBM Z and IBM LinuxONE with RHEL KVM --- # Creating a cluster with multi-architecture compute machines on IBM Z and IBM LinuxONE with RHEL KVM {#creating-multi-arch-compute-nodes-ibm-z-kvm} To create a cluster with multi-architecture compute machines on IBM Z(R) and IBM(R) LinuxONE (`s390x`) with RHEL KVM, you must have an existing single-architecture `x86_64` cluster. You can then add `s390x` compute machines to your OpenShift Container Platform cluster. Before you can add `s390x` nodes to your cluster, you must upgrade your cluster to one that uses the multi-architecture payload. For more information on migrating to the multi-architecture payload, see "Migrating to a cluster with multi-architecture compute machines". The following procedures explain how to create a RHCOS compute machine using a RHEL KVM instance. This will allow you to add `s390x` nodes to your cluster and deploy a cluster with multi-architecture compute machines. To create an IBM Z(R) or IBM(R) LinuxONE (`s390x`) cluster with multi-architecture compute machines on `x86_64`, follow the instructions for "Installing a cluster on IBM Z(R) and IBM(R) LinuxONE". You can then add `x86_64` compute machines as described in "Creating a cluster with multi-architecture compute machines on bare metal, IBM Power, or IBM Z". > [!NOTE] > Before adding a secondary architecture node to your cluster, install the Multiarch Tuning Operator and then deploy a `ClusterPodPlacementConfig` object. For more information, see "Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator". ## Creating RHCOS machines using `virt-install` {#machine-user-infra-machines-ibm-z-kvm_creating-multi-arch-compute-nodes-ibm-z-kvm} You can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines for your cluster by using `virt-install`. **Prerequisites** - You have at least one LPAR running on RHEL 8.7 or later with KVM, referred to as RHEL KVM host in this procedure. - The KVM/QEMU hypervisor is installed on the RHEL KVM host. - You have a domain name server (DNS) that can perform hostname and reverse lookup for the nodes. - An HTTP or HTTPS server is set up. **Procedure** 1. Extract the Ignition config file from the cluster by running the following command: ```terminal $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign ``` 2. Upload the `worker.ign` Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file. 3. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node: ```terminal $ curl -k http:///worker.ign ``` 4. Download the RHEL live `kernel`, `initramfs`, and `rootfs` files by running the following commands: ```terminal $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \ | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location') ``` ```terminal $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \ | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location') ``` ```terminal $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \ | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location') ``` 5. Move the downloaded RHEL live `kernel`, `initramfs`, and `rootfs` files to an HTTP or HTTPS server before you launch `virt-install`. 6. Create the new KVM guest nodes using the RHEL `kernel`, `initramfs`, and Ignition files; the new disk image; and adjusted parm line arguments. ```terminal $ virt-install \ --connect qemu:///system \ --name \ --autostart \ --os-variant rhel9.4 \ --cpu host \ --vcpus \ --memory \ --disk .qcow2,size= \ --network network= \ --location ,kernel=,initrd= \ --extra-args "rd.neednet=1" \ --extra-args "coreos.inst.install_dev=/dev/vda" \ --extra-args "coreos.inst.ignition_url=http:///worker.ign " \ --extra-args "coreos.live.rootfs_url=http:///rhcos--live-rootfs..img" \ --extra-args "ip=::::::none" \ --extra-args "nameserver=" \ --extra-args "console=ttysclp0" \ --noautoconsole \ --wait ``` where: `os-variant` : Specifies the RHEL version for the RHCOS compute machine. `rhel9.4` is the recommended version. To query the supported RHEL version of your operating system, run the following command: ```terminal $ osinfo-query os -f short-id ``` > [!NOTE] > The `os-variant` is case sensitive. `location` : Specifies the location of the kernel/initrd on the HTTP or HTTPS server. `coreos.inst.ignition_url` : Specifies the location of the `worker.ign` config file. Only HTTP and HTTPS protocols are supported. `coreos.live.rootfs_url` : Specifies the location of the `rootfs` artifact for the `kernel` and `initramfs` you are booting. Only HTTP and HTTPS protocols are supported `hostname` : Optional parameter. Specifies the fully qualified hostname of the client machine. > [!NOTE] > If you are using HAProxy as a load balancer, update your HAProxy rules for `ingress-router-443` and `ingress-router-80` in the `/etc/haproxy/haproxy.cfg` configuration file. 7. Continue to create more compute machines for your cluster. ## Approving the certificate signing requests for your machines {#installation-approve-csrs_creating-multi-arch-compute-nodes-ibm-z-kvm} To allow newly added machines to join your OpenShift Container Platform cluster, confirm that the cluster approves pending certificate signing requests (CSRs), or approve them yourself. Approve client requests first, then server requests. **Prerequisites** - You added machines to your cluster. **Procedure** 1. Confirm that the cluster recognizes the machines: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION master-0 Ready master 63m v1.35.4 master-1 Ready master 63m v1.35.4 master-2 Ready master 64m v1.35.4 ``` The output lists all of the machines that you created. > [!NOTE] > The preceding output might not include the compute nodes until you approve some CSRs. 2. Review the pending CSRs and ensure that you see the client requests with the `Pending` or `Approved` status for each machine that you added to the cluster: ```terminal $ oc get csr ``` ```terminal {title="Example output"} NAME AGE REQUESTOR CONDITION csr-8b2br 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending csr-8vnps 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending ... ``` In this example, two machines are joining the cluster. You might see more approved CSRs in the list. 3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in `Pending` status, approve the CSRs for your cluster machines: > [!NOTE] > You must approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates rotate, and more than two certificates are present for each node. You must approve all of these certificates. After you approve the client CSR, the kubelet creates a secondary CSR for the serving certificate, which requires manual approval. The `machine-approver` then automatically approves later serving certificate renewal requests if the kubelet requests a new certificate with the same parameters. > [!NOTE] > For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If you do not approve a request, the `oc exec`, `oc rsh`, and `oc logs` commands cannot succeed, because the API server requires a serving certificate when it connects to the kubelet. Any operation that contacts the kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the `node-bootstrapper` service account in the `system:node` or `system:admin` groups submitted the CSR, and confirm the identity of the node. - To approve them individually, run the following command for each valid CSR: ```terminal $ oc adm certificate approve ``` where: `` : Specifies the name of a CSR from the list of current CSRs. - To approve all pending CSRs, run the following command: ```terminal $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve ``` > [!NOTE] > Some Operators might not become available until you approve some CSRs. Each node submits two CSRs, so you might need to run the command to approve CSRs many times. 4. After you approve your client requests, review the server requests for each machine that you added to the cluster: ```terminal $ oc get csr ``` ```terminal {title="Example output"} NAME AGE REQUESTOR CONDITION csr-bfd72 5m26s system:node:ip-10-0-50-126.us-east-2.compute.internal Pending csr-c57lv 5m26s system:node:ip-10-0-95-157.us-east-2.compute.internal Pending ... ``` 5. If the remaining CSRs are not approved, and are in the `Pending` status, approve the CSRs for your cluster machines: - To approve them individually, run the following command for each valid CSR: ```terminal $ oc adm certificate approve ``` where: `` : Specifies the name of a CSR from the list of current CSRs. - To approve all pending CSRs, run the following command: ```terminal $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve ``` 6. After you approve all client and server CSRs, the machines have the `Ready` status. Verify this by running the following command: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION master-0 Ready master 73m v1.35.4 master-1 Ready master 73m v1.35.4 master-2 Ready master 74m v1.35.4 worker-0 Ready worker 11m v1.35.4 worker-1 Ready worker 11m v1.35.4 ``` > [!NOTE] > You might need to wait a few minutes after approval of the server CSRs for the machines to change to the `Ready` status. **Additional resources** {._additional-resources} - [Migrating to a cluster with multi-architecture compute machines](/openshift-docs-markdown/updating/updating_a_cluster/migrating-to-multi-payload#migrating-to-multi-payload) - [Installing a cluster on IBM Z(R) and IBM(R) LinuxONE](/openshift-docs-markdown/installing/installing_ibm_z/preparing-to-install-on-ibm-z#preparing-to-install-on-ibm-z) - [Creating a cluster with multi-architecture compute machines on bare metal, IBM Power, or IBM Z](/openshift-docs-markdown/post_installation_configuration/configuring-multi-arch-compute-machines/creating-multi-arch-compute-nodes-bare-metal#creating-multi-arch-compute-nodes-bare-metal) - [Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator](/openshift-docs-markdown/post_installation_configuration/configuring-multi-arch-compute-machines/multiarch-tuning-operator#multiarch-tuning-operator)