Creating a cluster with multi-architecture compute machines on Azure

To deploy a cluster on Microsoft Azure with multi-architecture compute machines, you must first create a single-architecture installer-provisioned cluster that uses the multi-architecture installer binary.

You can also migrate your current cluster with single-architecture compute machines to a cluster with multi-architecture compute machines. After creating a multi-architecture cluster, you can add nodes with different architectures to the cluster.

Creating a 64-bit ARM boot image using the Azure image gallery

You can generate a 64-bit x86 boot image or a 64-bit ARM boot image by using the Azure image gallery.

The procedure example describes how to manually generate a 64-bit ARM boot image. If you want to generate a 64-bit x86 boot image, replace aarch64 with x86_64; Additionally, replace any instances of rhcos-arm64 with rhcos-x86_64.

Prerequisites

• You installed the Azure CLI (az).
• You created a single-architecture Azure installer-provisioned cluster with the multi-architecture installer binary.

Procedure

1. Log in to your Azure account by running the following command:

   $ az login

2. Create a storage account and upload the aarch64 virtual hard drive (VHD) to your storage account. The OpenShift Container Platform installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:

   $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS

   • The westus object is an example region.

3. Create a storage container using the storage account you generated by entering the following command:

   $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

4. You must use the OpenShift Container Platform installation program JSON file to extract the URL and aarch64 VHD name:

   1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')

      • For a 64-bit x86 boot image, replace architectures.aarch64 with architectures.x86_64.

   2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd

      • For a 64-bit x86 boot image, replace architectures.aarch64 with architectures.x86_64 and replace aarch64.vhd with x86_64.vhd.

5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container with the following commands:

   $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

   $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

6. Copy the RHCOS VHD into the storage container:

   $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

   You can check the status of the copying process with the following command:

   $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

   Example output

   {
    "completionTime": null,
    "destinationSnapshot": null,
    "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    "incrementalCopy": null,
    "progress": "17179869696/17179869696",
    "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    "status": "success",
    "statusDescription": null
   }

   If the status parameter displays the success object, the copying process is complete.

7. Create an image gallery using the following command:

   $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

   Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

   $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2

   • For a 64-bit x86 boot image, replace --offer arm --sku arm64 with --offer x86_64 --sku x86_64 .

8. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

   $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

9. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.

   $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

10. Optional: Now that your arm64 boot image is now generated, you can access the ID of your image with the following command:

    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID

    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0

Adding a multi-architecture compute machine set to your Azure cluster

After creating a multi-architecture cluster, you can add nodes with different architectures.

You can add multi-architecture compute machines to a multi-architecture cluster in the following ways:

• Adding 64-bit x86 compute machines to a cluster that uses 64-bit ARM control plane machines and already includes 64-bit ARM compute machines. In this case, 64-bit x86 is considered the secondary architecture.
• Adding 64-bit ARM compute machines to a cluster that uses 64-bit x86 control plane machines and already includes 64-bit x86 compute machines. In this case, 64-bit ARM is considered the secondary architecture.

To create a custom compute machine set on Azure, see "Creating a compute machine set on Azure".

note

Before adding a secondary architecture node to your cluster, it is recommended to install the Multiarch Tuning Operator, and deploy a ClusterPodPlacementConfig custom resource. For more information, see "Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator".

Prerequisites

• You installed the OpenShift CLI (oc).
• You created a 64-bit ARM or 64-bit x86 boot image.
• You used the installation program to create a 64-bit ARM or 64-bit x86 single-architecture cluster with the multi-architecture installer binary.

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for an Azure 64-bit ARM or x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
       machine.openshift.io/cluster-api-machine-role: worker
       machine.openshift.io/cluster-api-machine-type: worker
     name: <infrastructure_id>-machine-set-0
     namespace: openshift-machine-api
   spec:
     replicas: 2
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: worker
           machine.openshift.io/cluster-api-machine-type: worker
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
       spec:
         lifecycleHooks: {}
         metadata: {}
         providerSpec:
           value:
             acceleratedNetworking: true
             apiVersion: machine.openshift.io/v1beta1
             credentialsSecret:
               name: azure-cloud-credentials
               namespace: openshift-machine-api
             image:
               offer: ""
               publisher: ""
               resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0
               sku: ""
               version: ""
             kind: AzureMachineProviderSpec
             location: <region>
             managedIdentity: <infrastructure_id>-identity
             networkResourceGroup: <infrastructure_id>-rg
             osDisk:
               diskSettings: {}
               diskSizeGB: 128
               managedDisk:
                 storageAccountType: Premium_LRS
               osType: Linux
             publicIP: false
             publicLoadBalancer: <infrastructure_id>
             resourceGroup: <infrastructure_id>-rg
             subnet: <infrastructure_id>-worker-subnet
             userDataSecret:
               name: worker-user-data
             vmSize: Standard_D4ps_v5
             vnet: <infrastructure_id>-vnet
             zone: "<zone>"

   where:

   image.resourceID

   Specifies the boot image, such as arm64 or amd64.

   vmSize

   Specifies the instance type used in your installation. Some example instance types are Standard_D4ps_v5 or D8ps.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name>

   • Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: arm64-machine-set-0.yaml, or amd64-machine-set-0.yaml.

Verification

1. Verify that the new machines are running by running the following command:

   $ oc get machineset -n openshift-machine-api

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-machine-set-0                   2        2      2          2  10m

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

Additional resources

• Installing a cluster on Azure with customizations
• Migrating to a cluster with multi-architecture compute machines
• Creating a compute machine set on Azure
• Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator