Installing a cluster on Azure into a government region¶

In OpenShift Container Platform version 4.22, you can install a cluster on Microsoft Azure into a government region. To configure the government region, you modify parameters in the install-config.yaml file before you install the cluster.

Azure government regions¶

Microsoft Azure Government (MAG) is a cloud environment designed for US government agencies for US government agencies at the federal, state, and local level, as well as contractors, educational institutions, and other US customers that must run sensitive workloads on Azure. MAG is composed of government-only data center regions, all granted an Impact Level 5 Provisional Authorization.

Deploying a OpenShift Container Platform cluster to a Microsoft Azure Government (MAG) region requires manually configuring the Azure Government cloud instance and region in the install-config.yaml file before you install the cluster. You must also update your service principal to reference the appropriate government environment.

Private clusters¶

You can deploy a private OpenShift Container Platform cluster that does not expose external endpoints. Private clusters are accessible from only an internal network and are not visible to the internet.

By default, OpenShift Container Platform is provisioned to use publicly-accessible DNS and endpoints. A private cluster sets the DNS, Ingress Controller, and API server to private when you deploy your cluster. This means that the cluster resources are only accessible from your internal network and are not visible to the internet.

To deploy a private cluster, you must:

• Use existing networking that meets your requirements. Your cluster resources might be shared between other clusters on the network.
• Deploy from a machine that has access to: ** The API services for the cloud to which you provision. ** The hosts on the network that you provision. ** The internet to obtain installation media.

Private clusters in Azure¶

To create a private cluster on Microsoft Azure, you must provide an existing private VNet and subnets to host the cluster. The installation program must also be able to resolve the DNS records that the cluster requires. The installation program configures the Ingress Operator and API server for only internal traffic.

Depending how your network connects to the private VNET, you might need to use a DNS forwarder to resolve the cluster’s private DNS records. The cluster’s machines use 168.63.129.16 internally for DNS resolution. For more information, see "What is Azure Private DNS?" and "What is IP address 168.63.129.16?".

The cluster still requires access to internet to access the Azure APIs.

The following items are not required or created when you install a private cluster:

• A BaseDomainResourceGroup, since the cluster does not create public records
• Public IP addresses
• Public DNS records
• Public endpoints

  The cluster is configured so that the Operators do not create public records for the cluster and all cluster machines are placed in the private subnets that you specify.

Limitations¶

Private clusters on Azure are subject to only the limitations that are associated with the use of an existing VNet.

User-defined outbound routing¶

You can configure user-defined outbound routing for a OpenShift Container Platform cluster to reach the internet without public IP addresses or a public load balancer.

You must use a pre-existing VNet for outbound routing when you install a cluster. The installation program does not configure this network.

When you configure a cluster to use user-defined routing, the installation program does not create the following resources:

• Outbound rules for access to the internet.
• Public IPs for the public load balancer.
• Kubernetes Service object to add the cluster machines to the public load balancer for outbound requests.

Before you set user-defined routing, ensure that the following items are available:

• Egress to the internet so that you can pull container images, unless you use an OpenShift image registry mirror.
• Access from the cluster to Azure APIs.
• Access to the required allowlist endpoints.

Several pre-existing networking setups support internet access by using user-defined routing.

About reusing a VNet for your OpenShift Container Platform cluster¶

In OpenShift Container Platform 4.22, you can deploy a cluster into an existing Microsoft Azure Virtual Network (VNet). Deployments in an existing VNet require existing subnets and routing rules.

By deploying OpenShift Container Platform into an existing Azure VNet, you might be able to avoid service limit constraints in new accounts or more easily abide by the operational constraints that your company’s guidelines set. This is a good option to use if you cannot obtain the infrastructure creation permissions that are required to create the VNet.

Requirements for using your VNet¶

When you deploy a cluster by using an existing VNet, you must perform additional network configuration before you install the cluster. In installer-provisioned infrastructure clusters, the installation program usually creates the following components, but it does not create them when you install into an existing VNet:

• Subnets
• Route tables
• VNets
• Network Security Groups

If you use a custom VNet, you must correctly configure it and its subnets for the installation program and the cluster to use. The installation program cannot subdivide network ranges for the cluster to use, set route tables for the subnets, or set VNet options like DHCP, so you must do so before you install the cluster.

The cluster must be able to access the resource group that contains the existing VNet and subnets. While all of the resources that the cluster creates are placed in a separate resource group that it creates, some network resources are used from a separate group. Some cluster Operators must be able to access resources in both resource groups. For example, the Machine API controller attaches NICs for the virtual machines that it creates to subnets from the networking resource group.

Your VNet must meet the following characteristics:

• The VNet’s CIDR block must contain the Networking.MachineCIDR range, which is the IP address pool for cluster machines.
• The VNet and its subnets must belong to the same resource group, and the subnets must be configured to use Azure-assigned DHCP IP addresses instead of static IP addresses.

You must provide two subnets within your VNet, one for the control plane machines and one for the compute machines. Because Azure distributes machines in different availability zones within the region that you specify, your cluster will have high availability by default.

To ensure that the subnets that you provide are suitable, the installation program confirms the following data:

• All the specified subnets exist.
• There are two private subnets, one for the control plane machines and one for the compute machines.
• The subnet CIDRs belong to the machine CIDR that you specified. Machines are not provisioned in availability zones that you do not provide private subnets for. If required, the installation program creates public load balancers that manage the control plane and worker nodes, and Azure allocates a public IP address to them.

Network security group requirements¶

The network security groups for the subnets that host the compute and control plane machines require specific access to ensure that the cluster communication is correct. You must create rules to allow access to the required cluster communication ports.

Required ports

1. If you are using Azure Firewall to restrict the internet access, then you can configure Azure Firewall to allow the Azure APIs. A network security group rule is not needed. For more information, see "Configuring your firewall" in "Additional resources".

Because cluster components do not modify the user-provided network security groups, which the Kubernetes controllers update, a pseudo-network security group is created for the Kubernetes controller to modify without impacting the rest of the environment.

Ports used for all-machine to all-machine communications

Ports used for control plane machine to control plane machine communications

Division of permissions¶

Starting with OpenShift Container Platform 4.3, you do not need all of the permissions that are required for an installation program-provisioned infrastructure cluster to deploy a cluster. This change mimics the division of permissions that you might have at your company: some individuals can create different resources in your clouds than others. For example, you might be able to create application-specific items, like instances, storage, and load balancers, but not networking-related components such as VNets, subnet, or ingress rules.

The Azure credentials that you use when you create your cluster do not need the networking permissions that are required to make VNets and core networking components within the VNet, such as subnets, routing tables, internet gateways, NAT, and VPN. You still need permission to make the application resources that the machines within the cluster require, such as load balancers, security groups, storage accounts, and nodes.

Isolation between clusters¶

Because the cluster is unable to modify network security groups in an existing subnet, there is no way to isolate clusters from each other on the VNet.

Manually creating the installation configuration file¶

Installing the cluster requires that you manually create the installation configuration file.

Prerequisites

• You have an SSH public key on your local machine for use with the installation program. You can use the key for SSH authentication onto your cluster nodes for debugging and disaster recovery.
• You have obtained the OpenShift Container Platform installation program and the pull secret for your cluster.

Procedure

1. Create an installation directory to store your required installation assets in:
   terminal

   $ mkdir <installation_directory>

   Important

   You must create a directory. Some installation assets, such as bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

2. Customize the provided sample install-config.yaml file template and save the file in the <installation_directory>.
   1. Edit the install-config.yaml file so that the value of the platform.azure.cloudName parameter is AzureUSGovernmentCloud.
   Note

   You must name this configuration file install-config.yaml.

3. Back up the install-config.yaml file so that you can use it to install many clusters.
   Important

   Back up the install-config.yaml file now, because the installation process consumes the file in the next step.

Minimum resource requirements for cluster installation¶

To ensure that your OpenShift Container Platform cluster runs as expected, each cluster machine must meet minimum CPU, memory, and storage requirements.

Minimum resource requirements

• One vCPU is equal to one physical core when simultaneous multithreading (SMT), or Hyper-Threading, is not enabled. When enabled, use the following formula to calculate the corresponding ratio: (threads per core × cores) × sockets = vCPUs.
• OpenShift Container Platform and Kubernetes are sensitive to disk performance, and Red Hat recommends faster storage, particularly for etcd on the control plane nodes which require a 10 ms p99 fsync duration. On many cloud platforms, storage size and IOPS scale together, so you might need to provision more storage to get enough performance.
• As with all user-provisioned installations, if you choose to use RHEL compute machines in your cluster, you take responsibility for all operating system life cycle management and maintenance, including performing system updates, applying patches, and completing all other required tasks. OpenShift Container Platform 4.10 and later do not support RHEL 7 compute machines.

If an instance type for your platform meets the minimum requirements for cluster machines, it is supported to use in OpenShift Container Platform.

Tested instance types for Azure¶

There are several Microsoft Azure instance types tested with OpenShift Container Platform. Choose a listed instance type when you install a cluster on 64-bit x86 infrastructure.

See the following machine types based on 64-bit x86 architecture:

General Purpose¶

Memory Optimized¶

Compute Optimized¶

Storage Optimized¶

GPU Accelerated¶

FPGA Accelerated¶

High Performance Compute¶

Enabling trusted launch for Azure VMs¶

To enable trusted launch on Azure virtual machines for your OpenShift Container Platform cluster, you can configure secure boot and virtualized Trusted Platform Modules in the install-config.yaml file. Apply the settings to control plane nodes, compute nodes, or all nodes as needed.

For more information about the sizes of virtual machines that support the trusted launch features, secure boot, and virtualized Trusted Platform Modules, see the Additional resources section.

Prerequisites

• You have created an install-config.yaml file.

Procedure

• Edit the install-config.yaml file before deploying your cluster:
  • Enable trusted launch only on control plane by adding the following stanza:
    yaml

    controlPlane:
      platform:
        azure:
          settings:
            securityType: TrustedLaunch
            trustedLaunch:
              uefiSettings:
                secureBoot: Enabled
                virtualizedTrustedPlatformModule: Enabled

  • Enable trusted launch only on compute node by adding the following stanza:
    yaml

    compute:
      platform:
        azure:
          settings:
            securityType: TrustedLaunch
            trustedLaunch:
              uefiSettings:
                secureBoot: Enabled
                virtualizedTrustedPlatformModule: Enabled

  • Enable trusted launch on all nodes by adding the following stanza:
    yaml

    platform:
      azure:
        settings:
          securityType: TrustedLaunch
          trustedLaunch:
            uefiSettings:
              secureBoot: Enabled
              virtualizedTrustedPlatformModule: Enabled

Enabling confidential VMs¶

To enable confidential VMs on Azure for your OpenShift Container Platform cluster, you can configure the install-config.yaml file before deployment. Apply the settings to control plane nodes, compute nodes, or all nodes as needed.

You can use confidential VMs with the following VM sizes:

• DCasv5-series
• DCadsv5-series
• ECasv5-series
• ECadsv5-series
• DCesv5-series
• DCedsv5-series
• ECesv5-series
• ECedsv5-series
• NCCads_H100_v5

Prerequisites

• You have created an install-config.yaml file.

Procedure

• Edit the install-config.yaml file before deploying your cluster:
  • Enable confidential VMs only on control plane by adding the following stanza:
    yaml

    controlPlane:
      platform:
        azure:
          settings:
            securityType: ConfidentialVM
            confidentialVM:
              uefiSettings:
                secureBoot: Enabled
                virtualizedTrustedPlatformModule: Enabled
          osDisk:
            securityProfile:
              securityEncryptionType: VMGuestStateOnly

  • Enable confidential VMs only on compute nodes by adding the following stanza:
    yaml

    compute:
      platform:
        azure:
          settings:
            securityType: ConfidentialVM
            confidentialVM:
              uefiSettings:
                secureBoot: Enabled
                virtualizedTrustedPlatformModule: Enabled
          osDisk:
            securityProfile:
              securityEncryptionType: VMGuestStateOnly

  • Enable confidential VMs on all nodes by adding the following stanza:
    yaml

    platform:
      azure:
        defaultMachinePlatform:
          settings:
            securityType: ConfidentialVM
            confidentialVM:
              uefiSettings:
                secureBoot: Enabled
                virtualizedTrustedPlatformModule: Enabled
          osDisk:
            securityProfile:
              securityEncryptionType: VMGuestStateOnly

Sample customized install-config.yaml file for Azure¶

You can customize the install-config.yaml file to specify more details about your OpenShift Container Platform cluster’s platform or modify the values of the required parameters.

apiVersion: v1
baseDomain: example.com
pullSecret: '{"auths": ...}'
sshKey: ssh-ed25519 AAAA...
metadata:
  name: example-cluster
controlPlane:
  hyperthreading: Enabled
  name: master
  platform:
    azure:
      type: Standard_D8s_v3
  replicas: 3
compute:
- hyperthreading: Enabled
  name: worker
  platform:
    azure:
      type: Standard_D2s_v3
  replicas: 3
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
platform:
  azure:
    baseDomainResourceGroupName: example-basedomain-resourcegroup-name
    region: centralus

where:

controlPlane

Specifies parameters that apply to control plane machines.

compute

Specifies parameters that apply to compute machines.

networking

Specifies parameters that apply to the cluster networking configuration. If you do not provide networking values, the installation program provides default values.

platform

Specifies parameters that apply to the infrastructure platform that hosts the cluster.

Configuring the cluster-wide proxy during installation¶

Production environments can deny direct access to the internet and instead have an HTTP or HTTPS proxy available. You can configure a new OpenShift Container Platform cluster to use a proxy by configuring the proxy settings in the install-config.yaml file.

Prerequisites

• You have an existing install-config.yaml file.
• You have reviewed the sites that your cluster requires access to and determined whether any of them need to bypass the proxy. By default, the proxy handles all cluster egress traffic, including calls to hosting cloud provider APIs. You added sites to the Proxy object’s spec.noProxy field to bypass the proxy if necessary.
  Note

  The Proxy object status.noProxy field includes the values of the networking.machineNetwork[].cidr, networking.clusterNetwork[].cidr, and networking.serviceNetwork[] fields from your installation configuration.

  For installations on Amazon Web Services (AWS), Google Cloud, Microsoft Azure, and Red Hat OpenStack Platform (RHOSP), the Proxy object status.noProxy field also includes the instance metadata endpoint (169.254.169.254).

Procedure

1. Edit your install-config.yaml file and add the proxy settings. For example:
   yaml

   apiVersion: v1
   baseDomain: my.domain.com
   proxy:
     httpProxy: http://<username>:<pswd>@<ip>:<port>
     httpsProxy: https://<username>:<pswd>@<ip>:<port>
     noProxy: example.com
   additionalTrustBundle: |
       -----BEGIN CERTIFICATE-----
       <MY_TRUSTED_CA_CERT>
       -----END CERTIFICATE-----
   additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle>
   # ...

   where:

   proxy.httpProxy

   Specifies a proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.

   proxy.httpsProxy

   Specifies a proxy URL to use for creating HTTPS connections outside the cluster.

   proxy.noProxy

   Specifies a comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass the proxy for all destinations.

   additionalTrustBundle

   If you specify this value, the installation program generates a config map named user-ca-bundle in the openshift-config namespace to hold the additional CA certificates. If you specify additionalTrustBundle and at least one proxy setting, the Proxy object references the user-ca-bundle config map in the trustedCA field. The Cluster Network Operator then creates a trusted-ca-bundle config map that merges the contents specified for the trustedCA parameter with the RHCOS trust bundle. You must set the additionalTrustBundle field unless an authority from the RHCOS trust bundle signs the proxy’s identity certificate.

   additionalTrustBundlePolicy

   Specifies the policy that determines the configuration of the Proxy object to reference the user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and Always. Use Proxyonly to reference the user-ca-bundle config map only when you configure an http/https proxy. Use Always to always reference the user-ca-bundle config map. The default value is Proxyonly. Optional parameter.

   Note

   The installation program does not support the proxy readinessEndpoints field.

   Note

   If the installation program times out, restart and then complete the deployment by using the wait-for command of the installation program. For example:

   terminal

   $ ./openshift-install wait-for install-complete --log-level debug

2. Save the file and reference it when installing OpenShift Container Platform.

   The installation program creates a cluster-wide proxy named cluster that uses the proxy settings in the install-config.yaml file. If you do not give proxy settings, the installation program still creates a cluster Proxy object, but it has a nil spec.

   Note

   Only the Proxy object named cluster is supported, and you cannot create additional proxies.

Deploying the cluster¶

To deploy your OpenShift Container Platform cluster, you initialize installation by running the openshift-install create cluster command from the directory that contains the installation program. The installation program provisions the required infrastructure and completes the cluster setup.

Prerequisites

• You have configured an account with the cloud platform that hosts your cluster.
• You have the OpenShift Container Platform installation program and the pull secret for your cluster.
• You have an Azure subscription ID and tenant ID.
• If you are installing the cluster using a service principal, you have its application ID and password.
• If you are installing the cluster using a system-assigned managed identity, you have enabled it on the virtual machine that you will run the installation program from.
• If you are installing the cluster using a user-assigned managed identity, you have met these prerequisites:
  • You have its client ID.
  • You have assigned it to the virtual machine that you will run the installation program from.

Procedure

. Optional: If you have run the installation program on this computer before, and want to use an alternative service principal or managed identity, go to the `~/.azure/` directory and delete the `osServicePrincipal.json` configuration file.

Deleting this file prevents the installation program from automatically reusing subscription and authentication values from a previous installation.

1. In the directory that contains the installation program, initialize the cluster deployment by running the following command:

$ ./openshift-install create cluster --dir <installation_directory> \
    --log-level=info

where:

• <installation_directory>: Specifies the location of your customized ./install-config.yaml file.
• --log-level: Specifies the log level. To view different installation details, specify warn, debug, or error instead of info.

If the installation program cannot locate the osServicePrincipal.json configuration file from a previous installation, you are prompted for Azure subscription and authentication values. . Enter the following Azure parameter values for your subscription: *** azure subscription id*: Enter the subscription ID to use for the cluster. *** azure tenant id*: Enter the tenant ID. . Depending on the Azure identity you are using to deploy the cluster, do one of the following when prompted for the azure service principal client id: ** If you are using a service principal, enter its application ID. ** If you are using a system-assigned managed identity, leave this value blank. ** If you are using a user-assigned managed identity, specify its client ID. . Depending on the Azure identity you are using to deploy the cluster, do one of the following when prompted for the azure service principal client secret: ** If you are using a service principal, enter its password. ** If you are using a system-assigned managed identity, leave this value blank. ** If you are using a user-assigned managed identity,leave this value blank.

Verification

When the cluster deployment completes successfully:

• The terminal displays directions for accessing your cluster, including a link to the web console and credentials for the kubeadmin user.
• Credential information also outputs to <installation_directory>/.openshift_install.log.
  Important

  Do not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

  The following example shows the expected output:

  terminal

  ...
  INFO Install complete!
  INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig'
  INFO Access the OpenShift web-console here: https://console-openshift-console.apps.mycluster.example.com
  INFO Login to the console with user: "kubeadmin", and password: "password"
  INFO Time elapsed: 36m22s

  Important

  • The Ignition config files that the installation program generates contain certificates that expire after 24 hours, which are then renewed at that time. If the cluster is shut down before renewing the certificates and the cluster is later restarted after the 24 hours have elapsed, the cluster automatically recovers the expired certificates. The exception is that you must manually approve the pending node-bootstrapper certificate signing requests (CSRs) to recover kubelet certificates. See the documentation for Recovering from expired control plane certificates for more information.
  • It is recommended that you use Ignition config files within 12 hours after they are generated because the 24-hour certificate rotates from 16 to 22 hours after the cluster is installed. By using the Ignition config files within 12 hours, you can avoid installation failure if the certificate update runs during installation.

Logging in to the cluster by using the CLI¶

To log in to your cluster as the default system user, export the kubeconfig file. This configuration enables the CLI to authenticate and connect to the specific API server created during OpenShift Container Platform installation.

The kubeconfig file is specific to a cluster and OpenShift Container Platform generates it during installation.

Prerequisites

• You deployed an OpenShift Container Platform cluster.
• You installed the OpenShift CLI (oc).

Procedure

1. Export the kubeadmin credentials by running the following command:
   terminal

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig

   where:

   <installation_directory>

   Specifies the path to the directory that stores the installation files.

2. Verify you can run oc commands successfully using the exported configuration by running the following command:
   terminal

   $ oc whoami

   Example output

   system:admin

Next steps

• "Customize your cluster"
• "Remote health reporting"