Creating a cluster with multi-architecture compute machines on Azure
Esc
Start typing to search...
On this page

Creating a cluster with multi-architecture compute machines on Azure

To deploy a cluster on Microsoft Azure with multi-architecture compute machines, you must first create a single-architecture installer-provisioned cluster that uses the multi-architecture installer binary.

You can also migrate your current cluster with single-architecture compute machines to a cluster with multi-architecture compute machines. After creating a multi-architecture cluster, you can add nodes with different architectures to the cluster.

Creating a 64-bit ARM boot image using the Azure image gallery

You can generate a 64-bit x86 boot image or a 64-bit ARM boot image by using the Azure image gallery.

The procedure example describes how to manually generate a 64-bit ARM boot image. If you want to generate a 64-bit x86 boot image, replace aarch64 with x86_64; Additionally, replace any instances of rhcos-arm64 with rhcos-x86_64.

Prerequisites

  • You installed the Azure CLI (az).
  • You created a single-architecture Azure installer-provisioned cluster with the multi-architecture installer binary.

Procedure

  1. Log in to your Azure account by running the following command:
    terminal
    $ az login
  2. Create a storage account and upload the aarch64 virtual hard drive (VHD) to your storage account. The OpenShift Container Platform installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:
    terminal
    $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS
    • The westus object is an example region.
  3. Create a storage container using the storage account you generated by entering the following command:
    terminal
    $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}
  4. You must use the OpenShift Container Platform installation program JSON file to extract the URL and aarch64 VHD name:
    1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:
      terminal
      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')
      • For a 64-bit x86 boot image, replace architectures.aarch64 with architectures.x86_64.
    2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:
      terminal
      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd
      • For a 64-bit x86 boot image, replace architectures.aarch64 with architectures.x86_64 and replace aarch64.vhd with x86_64.vhd.
  5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container with the following commands:
    terminal
    $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`
    terminal
    $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`
  6. Copy the RHCOS VHD into the storage container:
    terminal
    $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
 --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
 --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

    You can check the status of the copying process with the following command:

    terminal
    $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy
    Example output
    {
 "completionTime": null,
 "destinationSnapshot": null,
 "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
 "incrementalCopy": null,
 "progress": "17179869696/17179869696",
 "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
 "status": "success",
 "statusDescription": null
}

    If the status parameter displays the success object, the copying process is complete.

  7. Create an image gallery using the following command:
    terminal
    $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

    Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

    terminal
    $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2
    • For a 64-bit x86 boot image, replace --offer arm --sku arm64 with --offer x86_64 --sku x86_64 .
  8. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:
    terminal
    $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)
  9. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.
    terminal
    $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}
  10. Optional: Now that your arm64 boot image is now generated, you can access the ID of your image with the following command:
    terminal
    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID
    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0

Adding a multi-architecture compute machine set to your Azure cluster

After creating a multi-architecture cluster, you can add nodes with different architectures.

You can add multi-architecture compute machines to a multi-architecture cluster in the following ways:

  • Adding 64-bit x86 compute machines to a cluster that uses 64-bit ARM control plane machines and already includes 64-bit ARM compute machines. In this case, 64-bit x86 is considered the secondary architecture.
  • Adding 64-bit ARM compute machines to a cluster that uses 64-bit x86 control plane machines and already includes 64-bit x86 compute machines. In this case, 64-bit ARM is considered the secondary architecture.

To create a custom compute machine set on Azure, see "Creating a compute machine set on Azure".

Note

Before adding a secondary architecture node to your cluster, it is recommended to install the Multiarch Tuning Operator, and deploy a ClusterPodPlacementConfig custom resource. For more information, see "Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator".

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You created a 64-bit ARM or 64-bit x86 boot image.
  • You used the installation program to create a 64-bit ARM or 64-bit x86 single-architecture cluster with the multi-architecture installer binary.

Procedure

  1. Log in to the OpenShift CLI (oc).
  2. Create a YAML file and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.
    Example MachineSet object for an Azure 64-bit ARM or x86 compute node
    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
  name: <infrastructure_id>-machine-set-0
  namespace: openshift-machine-api
spec:
  replicas: 2
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
    spec:
      lifecycleHooks: {}
      metadata: {}
      providerSpec:
        value:
          acceleratedNetworking: true
          apiVersion: machine.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:
            offer: ""
            publisher: ""
            resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0
            sku: ""
            version: ""
          kind: AzureMachineProviderSpec
          location: <region>
          managedIdentity: <infrastructure_id>-identity
          networkResourceGroup: <infrastructure_id>-rg
          osDisk:
            diskSettings: {}
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          publicLoadBalancer: <infrastructure_id>
          resourceGroup: <infrastructure_id>-rg
          subnet: <infrastructure_id>-worker-subnet
          userDataSecret:
            name: worker-user-data
          vmSize: Standard_D4ps_v5
          vnet: <infrastructure_id>-vnet
          zone: "<zone>"

    where:

    image.resourceID

    Specifies the boot image, such as arm64 or amd64.

    vmSize

    Specifies the instance type used in your installation. Some example instance types are Standard_D4ps_v5 or D8ps.

  3. Create the compute machine set by running the following command:
    terminal
    $ oc create -f <file_name>
    • Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: arm64-machine-set-0.yaml, or amd64-machine-set-0.yaml.

Verification

  1. Verify that the new machines are running by running the following command:
    terminal
    $ oc get machineset -n openshift-machine-api

    The output must include the machine set that you created.

    Example output
    NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
<infrastructure_id>-machine-set-0                   2        2      2          2  10m
  2. You can check if the nodes are ready and schedulable by running the following command:
    terminal
    $ oc get nodes

Additional resources