--- title: Creating a cluster with multi-architecture compute machines on bare metal, IBM Power, or IBM Z --- # Creating a cluster with multi-architecture compute machines on bare metal, IBM Power, or IBM Z {#creating-multi-arch-compute-nodes-bare-metal} You can create a cluster with multi-architecture compute machines on bare metal (`x86_64` or `aarch64`), IBM Power(R) (`ppc64le`), or IBM Z(R) (`s390x`). To do this, you must have an existing single-architecture cluster on one of these platforms. See the following installation procedures for your platform: - Bare metal with user-provisioned infrastructure: See "Installing a user-provisioned cluster on bare metal". You can then add 64-bit ARM compute machines to your OpenShift Container Platform cluster on bare metal. - IBM Power(R): See "Installing on IBM Power(R)". You can then add `x86_64` compute machines to your OpenShift Container Platform cluster on IBM Power(R). - IBM Z(R) and IBM(R) LinuxONE: See "Installing on IBM Z(R) and IBM(R) LinuxONE". You can then add `x86_64` compute machines to your OpenShift Container Platform cluster on IBM Z(R) and IBM(R) LinuxONE. > [!IMPORTANT] > The bare metal installer-provisioned infrastructure and the Bare Metal Operator do not support adding secondary architecture nodes during the initial cluster setup. You can add secondary architecture nodes manually only after the initial cluster setup. Before you can add additional compute nodes to your cluster, you must upgrade your cluster to one that uses the multi-architecture payload. For more information about migrating to the multi-architecture payload, see "Migrating to a cluster with multi-architecture compute machines". The following procedures explain how to create a RHCOS compute machine by using an ISO image or network PXE booting. This allows you to add additional nodes to your cluster and deploy a cluster with multi-architecture compute machines. > [!NOTE] > Before adding a secondary architecture node to your cluster, you must install the Multiarch Tuning Operator, and deploy a `ClusterPodPlacementConfig` object. For more information, see "Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator". ## Creating RHCOS machines by using an ISO image {#machine-user-infra-machines-iso_creating-multi-arch-compute-nodes-bare-metal} To scale your OpenShift Container Platform bare metal cluster, you can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines by using an ISO image. **Prerequisites** - You have obtained the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation. - You must have the OpenShift CLI (`oc`) installed. **Procedure** 1. Extract the Ignition config file from the cluster by running the following command: ```terminal $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign ``` 2. Upload the `worker.ign` Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files. 3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node: ```terminal $ curl -k http:///worker.ign ``` 4. You can access the ISO image for booting your new machine by running the following command: ```terminal RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures..artifacts.metal.formats.iso.disk.location') ``` 5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster: - Burn the ISO image to a disk and boot it directly. - Use ISO redirection with a LOM interface. 6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment. > [!NOTE] > You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the `coreos-installer` command as outlined in the following steps, instead of adding kernel arguments. 7. Run the `coreos-installer` command by using `sudo`. The `core` user does not have the root privileges required to perform the installation. Specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to. ```terminal $ sudo coreos-installer install --ignition-url=http:///.ign --ignition-hash=sha512- ``` where: `` : Specifies the Ignition config file SHA512 digest obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. > [!NOTE] > If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running `coreos-installer`. The following example initializes a compute node installation to the `/dev/sda` device. The Ignition config file for the compute node is obtained from an HTTP web server with the IP address 192.168.1.2: ```terminal $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/worker.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b ``` 8. Monitor the progress of the RHCOS installation on the console of the machine. > [!IMPORTANT] > Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise. 9. Continue to create more compute machines for your cluster. ## Creating RHCOS machines by PXE or iPXE booting {#machine-user-infra-machines-pxe_creating-multi-arch-compute-nodes-bare-metal} To scale your OpenShift Container Platform bare metal cluster, you can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines by using PXE or iPXE booting. **Prerequisites** - You have obtained the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation. - You have obtained the URLs of the RHCOS ISO image, compressed metal BIOS, `kernel`, and `initramfs` files that you uploaded to your HTTP server during cluster installation. - You have access to the PXE booting infrastructure that you used to create the machines for your OpenShift Container Platform cluster during installation. The machines must boot from their local disks after RHCOS is installed on them. - If you use UEFI, you have access to the `grub.conf` file that you modified during OpenShift Container Platform installation. **Procedure** 1. Confirm that your PXE or iPXE installation for the RHCOS images is correct. - For PXE: ``` DEFAULT pxeboot TIMEOUT 20 PROMPT 0 LABEL pxeboot KERNEL http:///rhcos--live-kernel- APPEND initrd=http:///rhcos--live-initramfs..img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http:///worker.ign coreos.live.rootfs_url=http:///rhcos--live-rootfs..img ``` where: `KERNEL` : Specifies the location of the live `kernel` file that you uploaded to your HTTP server. `APPEND` : Specifies the locations of the RHCOS files that you uploaded to your HTTP server: `initrd` : Specifies the location of the live `initramfs` file. `coreos.inst.ignition_url` : Specifies the location of the worker Ignition config file. This parameter supports only HTTP and HTTPS. `coreos.live.rootfs_url` : Specifies the location of the live `rootfs` file. This parameter supports only HTTP and HTTPS. > [!NOTE] > This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more `console=` arguments to the `APPEND` line. For example, add `console=tty0 console=ttyS0` to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information on setting up a serial terminal and/or console in RHCOS, see "How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?". - For iPXE (`x86_64` + `aarch64`): ``` kernel http:///rhcos--live-kernel- initrd=main coreos.live.rootfs_url=http:///rhcos--live-rootfs..img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http:///worker.ign initrd --name main http:///rhcos--live-initramfs..img boot ``` where: `kernel` : Specifies the location of the `kernel` file that you uploaded to your HTTP server. `initrd=main` : Specifies an argument that is required for booting on UEFI systems. `coreos.live.rootfs_url` : Specifies the location of the `rootfs` file that you uploaded to your HTTP server. `coreos.inst.ignition_url` : Specifies the location of the worker Ignition config file that you uploaded to your HTTP server. `initrd --name main` : Specifies the location of the `initramfs` file that you uploaded to your HTTP server. > [!NOTE] > - If you use multiple NICs, specify a single interface in the `ip` option. For example, to use DHCP on a NIC named `eno1`, set `ip=eno1:dhcp`. > - This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more `console=` arguments to the `kernel` line. For example, add `console=tty0 console=ttyS0` to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information on setting up a serial terminal and/or console in RHCOS, see "How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?" in the Additional resources section and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section. > [!NOTE] > To network boot the CoreOS `kernel` on `aarch64` architecture, you need to use a version of iPXE build with the `IMAGE_GZIP` option enabled. For more information, see "IMAGE_GZIP option in iPXE". - For PXE (with UEFI and GRUB as second stage) on `aarch64`: ``` menuentry 'Install CoreOS' { linux rhcos--live-kernel- coreos.live.rootfs_url=http:///rhcos--live-rootfs..img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http:///worker.ign initrd rhcos--live-initramfs..img } ``` where: `linux` : Specifies the location of the live `kernel` file on your TFTP server. `coreos.live.rootfs_url` : Specifies the location of the live `rootfs` file. `coreos.inst.ignition_url` : Specifies the location of the worker Ignition config file. `initrd` : Specifies the location of the live `initramfs` file on your TFTP server. > [!NOTE] > If you use multiple NICs, specify a single interface in the `ip` option. For example, to use DHCP on a NIC named `eno1`, set `ip=eno1:dhcp`. 2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster. **Additional resources** {._additional-resources} - [How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? (Red Hat Knowledgebase article)](https://access.redhat.com/articles/7212) - [`IMAGE_GZIP` option in iPXE (iPXE documentation)](https://ipxe.org/buildcfg/image_gzip) ## Approving the certificate signing requests for your machines {#installation-approve-csrs_creating-multi-arch-compute-nodes-bare-metal} To allow newly added machines to join your OpenShift Container Platform cluster, confirm that the cluster approves pending certificate signing requests (CSRs), or approve them yourself. Approve client requests first, then server requests. **Prerequisites** - You added machines to your cluster. **Procedure** 1. Confirm that the cluster recognizes the machines: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION master-0 Ready master 63m v1.35.4 master-1 Ready master 63m v1.35.4 master-2 Ready master 64m v1.35.4 ``` The output lists all of the machines that you created. > [!NOTE] > The preceding output might not include the compute nodes until you approve some CSRs. 2. Review the pending CSRs and ensure that you see the client requests with the `Pending` or `Approved` status for each machine that you added to the cluster: ```terminal $ oc get csr ``` ```terminal {title="Example output"} NAME AGE REQUESTOR CONDITION csr-8b2br 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending csr-8vnps 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending ... ``` In this example, two machines are joining the cluster. You might see more approved CSRs in the list. 3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in `Pending` status, approve the CSRs for your cluster machines: > [!NOTE] > You must approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates rotate, and more than two certificates are present for each node. You must approve all of these certificates. After you approve the client CSR, the kubelet creates a secondary CSR for the serving certificate, which requires manual approval. The `machine-approver` then automatically approves later serving certificate renewal requests if the kubelet requests a new certificate with the same parameters. > [!NOTE] > For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If you do not approve a request, the `oc exec`, `oc rsh`, and `oc logs` commands cannot succeed, because the API server requires a serving certificate when it connects to the kubelet. Any operation that contacts the kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the `node-bootstrapper` service account in the `system:node` or `system:admin` groups submitted the CSR, and confirm the identity of the node. - To approve them individually, run the following command for each valid CSR: ```terminal $ oc adm certificate approve ``` where: `` : Specifies the name of a CSR from the list of current CSRs. - To approve all pending CSRs, run the following command: ```terminal $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve ``` > [!NOTE] > Some Operators might not become available until you approve some CSRs. Each node submits two CSRs, so you might need to run the command to approve CSRs many times. 4. After you approve your client requests, review the server requests for each machine that you added to the cluster: ```terminal $ oc get csr ``` ```terminal {title="Example output"} NAME AGE REQUESTOR CONDITION csr-bfd72 5m26s system:node:ip-10-0-50-126.us-east-2.compute.internal Pending csr-c57lv 5m26s system:node:ip-10-0-95-157.us-east-2.compute.internal Pending ... ``` 5. If the remaining CSRs are not approved, and are in the `Pending` status, approve the CSRs for your cluster machines: - To approve them individually, run the following command for each valid CSR: ```terminal $ oc adm certificate approve ``` where: `` : Specifies the name of a CSR from the list of current CSRs. - To approve all pending CSRs, run the following command: ```terminal $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve ``` 6. After you approve all client and server CSRs, the machines have the `Ready` status. Verify this by running the following command: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION master-0 Ready master 73m v1.35.4 master-1 Ready master 73m v1.35.4 master-2 Ready master 74m v1.35.4 worker-0 Ready worker 11m v1.35.4 worker-1 Ready worker 11m v1.35.4 ``` > [!NOTE] > You might need to wait a few minutes after approval of the server CSRs for the machines to change to the `Ready` status. **Additional resources** {._additional-resources} - [Installing a user provisioned cluster on bare metal](/openshift-docs-markdown/installing/installing_bare_metal/upi/installing-bare-metal#installing-bare-metal) - [Installing a cluster on IBM Power(R)](/openshift-docs-markdown/installing/installing_ibm_power/preparing-to-install-on-ibm-power#preparing-to-install-on-ibm-power) - [Installing a cluster on IBM Z(R) and IBM(R) LinuxONE](/openshift-docs-markdown/installing/installing_ibm_z/preparing-to-install-on-ibm-z#preparing-to-install-on-ibm-z) - [Migrating to a cluster with multi-architecture compute machines](/openshift-docs-markdown/updating/updating_a_cluster/migrating-to-multi-payload#migrating-to-multi-payload) - [Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator](/openshift-docs-markdown/post_installation_configuration/configuring-multi-arch-compute-machines/multiarch-tuning-operator#multiarch-tuning-operator)