Creating a cluster with multi-architecture compute machines on Azure¶

To deploy a cluster on Microsoft Azure with multi-architecture compute machines, you must first create a single-architecture installer-provisioned cluster that uses the multi-architecture installer binary.

You can also migrate your current cluster with single-architecture compute machines to a cluster with multi-architecture compute machines. After creating a multi-architecture cluster, you can add nodes with different architectures to the cluster.

Creating a 64-bit ARM boot image using the Azure image gallery¶

You can generate a 64-bit x86 boot image or a 64-bit ARM boot image by using the Azure image gallery.

The procedure example describes how to manually generate a 64-bit ARM boot image. If you want to generate a 64-bit x86 boot image, replace aarch64 with x86_64; Additionally, replace any instances of rhcos-arm64 with rhcos-x86_64.

Prerequisites

You installed the Azure CLI (az).
You created a single-architecture Azure installer-provisioned cluster with the multi-architecture installer binary.

Procedure

Log in to your Azure account by running the following command:
```
$ az login
```
Create a storage account and upload the aarch64 virtual hard drive (VHD) to your storage account. The OpenShift Container Platform installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:
```
$ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS
```
- The westus object is an example region.
Create a storage container using the storage account you generated by entering the following command:
```
$ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}
```
You must use the OpenShift Container Platform installation program JSON file to extract the URL and aarch64 VHD name:
1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:
```
$ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')
```
  - For a 64-bit x86 boot image, replace architectures.aarch64 with architectures.x86_64.
2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:
```
$ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd
```
  - For a 64-bit x86 boot image, replace architectures.aarch64 with architectures.x86_64 and replace aarch64.vhd with x86_64.vhd.

Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container with the following commands:

$ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

$ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

Copy the RHCOS VHD into the storage container:

$ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
 --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
 --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

You can check the status of the copying process with the following command:

$ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

Example output

{
 "completionTime": null,
 "destinationSnapshot": null,
 "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
 "incrementalCopy": null,
 "progress": "17179869696/17179869696",
 "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
 "status": "success",
 "statusDescription": null
}

If the status parameter displays the success object, the copying process is complete.

Create an image gallery using the following command:

$ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

$ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2

For a 64-bit x86 boot image, replace --offer arm --sku arm64 with --offer x86_64 --sku x86_64.

To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

$ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.

$ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

Optional: Now that your arm64 boot image is now generated, you can access the ID of your image with the following command:
```
$ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0
```
The following example image ID is used in the recourseID parameter of the compute machine set:
Example resourceID
```
/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0
```

Adding a multi-architecture compute machine set to your Azure cluster¶

After creating a multi-architecture cluster, you can add nodes with different architectures.

You can add multi-architecture compute machines to a multi-architecture cluster in the following ways:

Adding 64-bit x86 compute machines to a cluster that uses 64-bit ARM control plane machines and already includes 64-bit ARM compute machines. In this case, 64-bit x86 is considered the secondary architecture.
Adding 64-bit ARM compute machines to a cluster that uses 64-bit x86 control plane machines and already includes 64-bit x86 compute machines. In this case, 64-bit ARM is considered the secondary architecture.

To create a custom compute machine set on Azure, see "Creating a compute machine set on Azure".

Note

Before adding a secondary architecture node to your cluster, it is recommended to install the Multiarch Tuning Operator, and deploy a ClusterPodPlacementConfig custom resource. For more information, see "Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator".

Prerequisites

You installed the OpenShift CLI (oc).
You created a 64-bit ARM or 64-bit x86 boot image.
You used the installation program to create a 64-bit ARM or 64-bit x86 single-architecture cluster with the multi-architecture installer binary.

Procedure

Log in to the OpenShift CLI (oc).

Create a YAML file and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

Example MachineSet object for an Azure 64-bit ARM or x86 compute node

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
  name: <infrastructure_id>-machine-set-0
  namespace: openshift-machine-api
spec:
  replicas: 2
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
    spec:
      lifecycleHooks: {}
      metadata: {}
      providerSpec:
        value:
          acceleratedNetworking: true
          apiVersion: machine.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:
            offer: ""
            publisher: ""
            resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0
            sku: ""
            version: ""
          kind: AzureMachineProviderSpec
          location: <region>
          managedIdentity: <infrastructure_id>-identity
          networkResourceGroup: <infrastructure_id>-rg
          osDisk:
            diskSettings: {}
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          publicLoadBalancer: <infrastructure_id>
          resourceGroup: <infrastructure_id>-rg
          subnet: <infrastructure_id>-worker-subnet
          userDataSecret:
            name: worker-user-data
          vmSize: Standard_D4ps_v5
          vnet: <infrastructure_id>-vnet
          zone: "<zone>"

where:

image.resourceID: Specifies the boot image, such as arm64 or amd64.
vmSize: Specifies the instance type used in your installation. Some example instance types are Standard_D4ps_v5 or D8ps.

Create the compute machine set by running the following command:
```
$ oc create -f <file_name>
```
- Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: arm64-machine-set-0.yaml, or amd64-machine-set-0.yaml.

Verification

Verify that the new machines are running by running the following command:

$ oc get machineset -n openshift-machine-api

The output must include the machine set that you created.

Example output

NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
<infrastructure_id>-machine-set-0                   2        2      2          2  10m

You can check if the nodes are ready and schedulable by running the following command:
```
$ oc get nodes
```

Additional resources