Skip to content

Deploying hosted control planes on OpenShift Virtualization

With hosted control planes and OpenShift Virtualization, you can create OpenShift Container Platform clusters with worker nodes that are hosted by KubeVirt virtual machines.

Hosted control planes on OpenShift Virtualization provides several benefits:

  • Enhances resource usage by packing hosted control planes and hosted clusters in the same underlying bare-metal infrastructure
  • Separates hosted control planes and hosted clusters to provide strong isolation
  • Reduces cluster provision time by eliminating the bare-metal node bootstrapping process
  • Manages many releases under the same base OpenShift Container Platform cluster

The hosted control planes feature is enabled by default.

You can use the hosted control plane command-line interface, hcp, to create an OpenShift Container Platform hosted cluster. The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine Operator".

Additional resources

Prerequisites to deploy hosted control planes on OpenShift Virtualization

To create an OpenShift Container Platform cluster on OpenShift Virtualization, you must meet several prerequisites.

  • You have administrator access to an OpenShift Container Platform cluster, version 4.14 or later, specified in the KUBECONFIG environment variable.

  • The OpenShift Container Platform management cluster must have wildcard DNS routes enabled, as shown in the following command:

    $ oc patch ingresscontroller -n openshift-ingress-operator default \
  --type=json \
  -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'

  • The OpenShift Container Platform management cluster has OpenShift Virtualization, version 4.14 or later, installed on it. For more information, see "Installing OpenShift Virtualization using the web console".

  • The OpenShift Container Platform management cluster is on-premise bare metal.

  • The OpenShift Container Platform management cluster must be configured with OVNKubernetes as the default pod network Container Network Interface (CNI). Live migration is supported for nodes only if the CNI is OVN-Kubernetes.

  • The OpenShift Container Platform management cluster has a default storage class. For more information, see "Postinstallation storage configuration". The following example shows how to set a default storage class:

    $ oc patch storageclass ocs-storagecluster-ceph-rbd \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

  • When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using Logical Volume Manager storage".

  • On the OpenShift Container Platform cluster that hosts the OpenShift Virtualization virtual machines, you must use a ReadWriteMany (RWX) storage class so that live migration can be enabled.

  • You have a valid pull secret file for the quay.io/openshift-release-dev repository. For more information, see "Install OpenShift on any x86_64 platform with user-provisioned infrastructure".

  • You have installed the hosted control plane command-line interface.

  • You have configured a load balancer. For more information, see "Configuring MetalLB".

  • For optimal network performance, you are using a network maximum transmission unit (MTU) of 9000 or greater on the OpenShift Container Platform cluster that hosts the KubeVirt virtual machines. If you use a lower MTU setting, network latency and the throughput of the hosted pods are affected. Enable multiqueue on node pools only when the MTU is 9000 or greater.

    Warning

    You cannot change the MTU value for your cluster as a postinstallation task.

  • The multicluster engine Operator has at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported. For more information about the local-cluster, see "Advanced configuration" in the multicluster engine Operator documentation. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster

  • Ensure each hosted cluster has a cluster-wide unique name.

  • Do not use clusters as a hosted cluster name.

  • Do not create a hosted cluster in the namespace of a multicluster engine Operator managed cluster.

Additional resources

OpenShift Virtualization firewall and port requirements

Ensure that you meet the firewall and port requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters.

  • The kube-apiserver service runs on port 6443 by default and requires ingress access for communication between the control plane components.

    • If you use the NodePort publishing strategy, ensure that the node port that is assigned to the kube-apiserver service is exposed.
    • If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses.

  • If you use the NodePort publishing strategy, use a firewall rule for the ignition-server and Oauth-server settings.

  • The konnectivity agent, which establishes a reverse tunnel to allow bi-directional communication on the hosted cluster, requires egress access to the cluster API server address on port 6443. With that egress access, the agent can reach the kube-apiserver service.

    • If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443.
    • If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes.

  • If you change the default port of 6443, adjust the rules to reflect that change.

  • Ensure that you open any ports that are required by the workloads that run in the clusters.

  • Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary.

  • For production deployments, use a load balancer to simplify access through a single IP address.

Live migration for compute nodes

While the management cluster for hosted cluster virtual machines (VMs) is undergoing updates or maintenance, the hosted cluster VMs can be automatically live migrated to prevent disrupting hosted cluster workloads. As a result, the management cluster can be updated without affecting the availability and operation of the KubeVirt platform hosted clusters.

Warning

The live migration of KubeVirt VMs is enabled by default provided that the VMs use ReadWriteMany (RWX) storage for both the root volume and the storage classes that are mapped to the kubevirt-csi CSI provider.

You can verify that the VMs in a node pool are capable of live migration by checking the KubeVirtNodesLiveMigratable condition in the status section of a NodePool object.

In the following example, the VMs cannot be live migrated because RWX storage is not used.

Example configuration where VMs cannot be live migrated
    - lastTransitionTime: "2024-10-08T15:38:19Z"
      message: |
        3 of 3 machines are not live migratable
        Machine user-np-ngst4-gw2hz: DisksNotLiveMigratable: user-np-ngst4-gw2hz is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-gw2hz-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
        Machine user-np-ngst4-npq7x: DisksNotLiveMigratable: user-np-ngst4-npq7x is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-npq7x-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
        Machine user-np-ngst4-q5nkb: DisksNotLiveMigratable: user-np-ngst4-q5nkb is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-q5nkb-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
      observedGeneration: 1
      reason: DisksNotLiveMigratable
      status: "False"
      type: KubeVirtNodesLiveMigratable

In the next example, the VMs meet the requirements to be live migrated.

Example configuration where VMs can be live migrated
    - lastTransitionTime: "2024-10-08T15:38:19Z"
      message: "All is well"
      observedGeneration: 1
      reason: AsExpected
      status: "True"
      type: KubeVirtNodesLiveMigratable

While live migration can protect VMs from disruption in normal circumstances, events such as infrastructure node failure can result in a hard restart of any VMs that are hosted on the failed node. For live migration to be successful, the source node that a VM is hosted on must be working correctly.

When the VMs in a node pool cannot be live migrated, workload disruption might occur on the hosted cluster during maintenance on the management cluster. By default, the hosted control planes controllers try to drain the workloads that are hosted on KubeVirt VMs that cannot be live migrated before the VMs are stopped. Draining the hosted cluster nodes before stopping the VMs allows pod disruption budgets to protect workload availability within the hosted cluster.

Configuring MetalLB

Before you can create a hosted cluster on the KubeVirt platform, you must have the MetalLB load balancer configured.

Prerequisites

  • You have installed the MetalLB Operator. For more information, see "Installing the MetalLB Operator".

Procedure

  1. Create a MetalLB resource by saving the following sample YAML content in the configure-metallb.yaml file:

    apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system

  2. Apply the YAML content by entering the following command:

    $ oc apply -f configure-metallb.yaml
    Example output
    metallb.metallb.io/metallb created

  3. Create a IPAddressPool resource by saving the following sample YAML content in the create-ip-address-pool.yaml file:

    apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: metallb
  namespace: metallb-system
spec:
  addresses:
  - 192.168.216.32-192.168.216.122

    Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network.

  4. Apply the YAML content by entering the following command:

    $ oc apply -f create-ip-address-pool.yaml
    Example output
    ipaddresspool.metallb.io/metallb created

  5. Create a L2Advertisement resource by saving the following sample YAML content in the l2advertisement.yaml file:

    apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
   - metallb

  6. Apply the YAML content by entering the following command:

    $ oc apply -f l2advertisement.yaml
    Example output
    l2advertisement.metallb.io/metallb created

Additional resources

Hosted clusters with the KubeVirt platform

With OpenShift Container Platform 4.14 or later, you can create a hosted cluster with KubeVirt by using the command-line interface (CLI), the console, or by using external infrastructure.

Creating a hosted cluster with the KubeVirt platform by using the CLI

To create a hosted cluster on OpenShift Virtualization, you can use the hosted control plane command-line interface (CLI), hcp.

Warning

Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels.

Procedure

  1. Create a hosted cluster with the KubeVirt platform by entering the following command:

    $ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \
  --node-pool-replicas <node_pool_replica_count> \
  --pull-secret <path_to_pull_secret> \
  --memory <value_for_memory> \
  --cores <value_for_cpu> \
  --etcd-storage-class=<etcd_storage_class> \
  --arch <architecture_of_the_nodepool> \
  --release-image <ocp_release_image_for_the_cluster> \
  --image-content-sources <path_to_image_content_sources_file> \
  --disable-cluster-capabilities=<capability> \
  --enable-cluster-capabilities=<capability> \
  --additional-trust-bundle <path_to_ca_bundle_file>
    • --name defines the name of your hosted cluster, for example, my-hosted-cluster.
    • --node-pool-replicas defines the node pool replica count, for example, 3. You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created.
    • --pull-secret defines the path to your pull secret, for example, /user/name/pullsecret.
    • --memory defines a value for memory, for example, 6Gi.
    • --cores defines a value for CPU, for example, 2.
    • --etcd-storage-class defines the etcd storage class name, for example, lvm-storageclass.
    • --arch defines the architecture of the node pool, for example, s390x. The default is amd64.
    • --release-image defines the OpenShift Container Platform release image for the cluster, for example, quay.io/openshift-release-dev/ocp-release:4.20.14-multi. You can use the --release-image flag to set up the hosted cluster with a specific OpenShift Container Platform release.
    • --image-content-sources specifies the path to a file with image content sources.
    • --disable-cluster-capabilities specifies that you want to disable optional capabilities. This flag is optional. For more information, see "Capabilities for hosted clusters".
    • --enable-cluster-capabilities specifies that you want to enable optional capabilities in the hosted cluster. This flag is optional. For more information, see "Capabilities for hosted clusters".
    • --additional-trust-bundle specifies the path to a file with user CA bundle.

    A default node pool is created for the cluster with a specific number of virtual machine worker replicas according to the --node-pool-replicas flag.

  2. After a few moments, verify that the hosted control plane pods are running by entering the following command:

    $ oc -n clusters-<hosted-cluster-name> get pods
    Example output
    NAME                                                  READY   STATUS    RESTARTS   AGE
capi-provider-5cc7b74f47-n5gkr                        1/1     Running   0          3m
catalog-operator-5f799567b7-fd6jw                     2/2     Running   0          69s
certified-operators-catalog-784b9899f9-mrp6p          1/1     Running   0          66s
cluster-api-6bbc867966-l4dwl                          1/1     Running   0          66s
.
.
.
redhat-operators-catalog-9d5fd4d44-z8qqk              1/1     Running   0          66s

    A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned.

Verification

  • To check the status of the hosted cluster, see the corresponding HostedCluster resource by entering the following command:

    $ oc get --namespace clusters hostedclusters

    See the following example output, which illustrates a fully provisioned HostedCluster object:

    NAMESPACE   NAME                VERSION     KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
clusters    my-hosted-cluster   <4.x.0>     example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

    Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use.

Additional resources

Creating a hosted cluster with the KubeVirt platform by using external infrastructure

By default, the HyperShift Operator hosts both the control plane pods of the hosted cluster and the KubeVirt worker VMs within the same cluster. With the external infrastructure feature, you can place the worker node VMs on a separate cluster from the control plane pods.

  • The management cluster is the OpenShift Container Platform cluster that runs the HyperShift Operator and hosts the control plane pods for a hosted cluster.
  • The infrastructure cluster is the OpenShift Container Platform cluster that runs the KubeVirt worker VMs for a hosted cluster.
  • By default, the management cluster also acts as the infrastructure cluster that hosts VMs. However, for external infrastructure, the management and infrastructure clusters are different.

Warning

Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels.

Prerequisites

  • You must have a namespace on the external infrastructure cluster for the KubeVirt nodes to be hosted in.
  • You must have a kubeconfig file for the external infrastructure cluster.

Procedure

  • In the hcp command-line interface, place the KubeVirt worker VMs on the infrastructure cluster, use the --infra-kubeconfig-file and --infra-namespace arguments, as shown in the following example:

    $ hcp create cluster kubevirt \
  --name <hosted-cluster-name> \
  --node-pool-replicas <worker-count> \
  --pull-secret <path-to-pull-secret> \
  --memory <value-for-memory> \
  --cores <value-for-cpu> \
  --infra-namespace=<hosted-cluster-namespace>-<hosted-cluster-name> \
  --infra-kubeconfig-file=<path-to-external-infra-kubeconfig>
    • --name defines the name of your hosted cluster, for example, my-hosted-cluster.
    • --node-pool-replicas defines the worker count, for example, 2.
    • --pull-secret defines the path to your pull secret, for example, /user/name/pullsecret.
    • --memory defines a value for memory, for example, 6Gi.
    • --cores defines a value for CPU, for example, 2.
    • --infra-namespace defines the infrastructure namespace, for example, clusters-example.
    • --infra-kubeconfig-file defines the path to your kubeconfig file for the infrastructure cluster, for example, /user/name/external-infra-kubeconfig.

    After you enter the command, the control plane pods are hosted on the management cluster that the HyperShift Operator runs on, and the KubeVirt VMs are hosted on a separate infrastructure cluster.

Additional resources

Creating a hosted cluster by using the console

If you prefer to work in the OpenShift Container Platform console instead of the CLI, you can create a hosted cluster on the KubeVirt platform by using the console.

Note

If you want to use predefined values to automatically populate fields in the console, you can create a OpenShift Virtualization credential. For more information, see "Creating a credential for an on-premises environment".

Procedure

  1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials.

  2. In the console header, ensure that All Clusters is selected.

  3. Click Infrastructure > Clusters.

  4. Click Create cluster > Red Hat OpenShift Virtualization > Hosted.

  5. On the Create cluster page, follow the prompts to enter details about the cluster and node pools.

    On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a OpenShift Virtualization credential, the pull secret is automatically populated.

    Warning

    Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels.

  6. On the Node pools page, expand the Networking options section and configure the networking options for your node pool:

    1. In the Additional networks field, enter a network name in the format of <namespace>/<name>; for example, my-namespace/network1. The namespace and the name must be valid DNS labels. Multiple networks are supported.
    2. By default, the Attach default pod network checkbox is selected. You can clear this checkbox only if additional networks exist.

  7. Review your entries and click Create.

    The Hosted cluster view is displayed.

Verification

  1. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name.
  2. Wait until the control plane components are ready. This process can take a few minutes.
  3. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Additional resources

Capabilities for hosted clusters

To reduce resource consumption and prevent unnecessary Operators and operands from being deployed, administrators can enable or disable optional OpenShift Container Platform components when they create a hosted cluster.

When capabilities are not specified on a HostedCluster resource, the cluster uses the OpenShift Container Platform version’s DefaultCapabilitySet settings, excluding the baremetal capability. As a result, most optional components are enabled by default.

Warning

Capabilities are immutable after cluster creation. You cannot change them after you create the HostedCluster resource.

Capabilities that you can enable or disable for a hosted cluster

Familiarize yourself with the supported capabilities that you can enable or disable for a HostedCluster resource.

The capabilities are described in the following table:

Capability Description
ImageRegistry The OpenShift Image Registry Operator and its operands, including cloud storage infrastructure, such as S3 buckets and Identity and Access Management (IAM) users.
openshift-samples The OpenShift Samples Operator, which manages example ImageStreams and templates.
Insights The Insights Operator, which collects and uploads cluster telemetry data.
baremetal The Bare Metal Infrastructure Operator. This capability is excluded from the default set. If needed, you must explicitly enable it.
Console The OpenShift Web Console Operator and its operands.
NodeTuning The Node Tuning Operator, which manages node-level performance tuning by using TuneD and performance profiles.
Ingress The OpenShift Ingress Operator, which manages the default router of the cluster.

The following rules apply when you combine capability settings:

No overlap
A capability cannot be in both the enabled and disabled lists simultaneously.
Console requires Ingress
You can disable the Ingress capability only if the Console capability is also disabled because the console depends on Ingress.
Version requirement
You must use OpenShift Container Platform 4.20 or later to disable any of the following capabilities: openshift-samples, Insights, Console, NodeTuning, and Ingress. You can disable ImageRegistry and baremetal on versions earlier than 4.20.
Bare metal default exclusion
The baremetal capability is excluded from the default set. You can add it to the cluster by explicitly enabling it.

Setting capabilities for a hosted cluster

To reduce unnecessary resource consumption, you can control which optional capabilities are enabled for a hosted cluster when you create the cluster.

Warning

Capabilities are immutable after cluster creation. You cannot change them after you create the HostedCluster resource.

You can specify which capabilities are enabled by either using the hcp command-line interface (CLI) or by setting the HostedCluster manifest.

Procedure

  • To specify which capabilities are enabled in a hosted cluster by using the CLI, you can add the --disable-cluster-capabilities flag, the --enable-cluster-capabilities flag, or both. The following example shows how to disable the ImageRegistry, Console, and Ingress capabilities and enable the baremetal capability while you create a hosted cluster on AWS by using the hcp command-line interface:

    $ hcp create cluster aws \
    --name my-hosted-cluster \
    --disable-cluster-capabilities=ImageRegistry,Console,Ingress \
    --enable-cluster-capabilities=baremetal

    You can specify multiple capabilities as a comma-separated list. The supported values are as follows:

    • ImageRegistry
    • openshift-samples
    • Insights
    • baremetal
    • Console
    • NodeTuning
    • Ingress

  • To specify which capabilities are enabled in a hosted cluster by using the HostedCluster manifest at cluster creation time, see the following examples:

    • To directly disable capabilities in a hosted cluster, add the spec.capabilities.disabled section in the HostedCluster resource:

      apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: my-hosted-cluster
  namespace: my-cluster-namespace
spec:
  capabilities:
    disabled:
      - ImageRegistry
      - Console
      - Ingress
  # ...

    • To explicitly enable a capability that is not part of the default set of capabilities, such as the baremetal capability, see the following example:

      apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: my-hosted-cluster
  namespace: my-cluster-namespace
spec:
  capabilities:
    enabled:
      - baremetal
  # ...

    • You can use both enabled and disabled if no capabilities are in both lists. See the following example:

      apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: my-hosted-cluster
  namespace: my-cluster-namespace
spec:
  capabilities:
    enabled:
      - baremetal
    disabled:
      - ImageRegistry
      - openshift-samples
  # ...

Configuring the default ingress and DNS for hosted control planes on OpenShift Virtualization

Every OpenShift Container Platform cluster includes a default application Ingress Controller, which must have an wildcard DNS record associated with it.

By default, hosted clusters that are created by using OpenShift Virtualization automatically become a subdomain of the OpenShift Container Platform cluster that the virtual machines run on.

For example, your OpenShift Container Platform cluster might have the following default ingress DNS entry:

*.apps.mgmt-cluster.example.com

As a result, a hosted cluster that is named guest and that runs on that underlying OpenShift Container Platform cluster has the following default ingress:

*.apps.guest.apps.mgmt-cluster.example.com

Procedure

  • For the default ingress DNS to work properly, the cluster that hosts the virtual machines must allow wildcard DNS routes. You can configure this behavior by entering the following command:

    $ oc patch ingresscontroller -n openshift-ingress-operator default \
  --type=json \
  -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'

    Note

    When you use the default hosted cluster ingress, connectivity is limited to HTTPS traffic over port 443. Plain HTTP traffic over port 80 is rejected. This limitation applies to only the default ingress behavior.

Defining a custom DNS name

As a cluster administrator, you can create a hosted cluster with an external API DNS name that differs from the internal endpoint that gets used for node bootstraps and control plane communication.

You might want to define a different DNS name for the following reasons:

  • To replace the user-facing TLS certificate with one from a public CA without breaking the control plane functions that bind to the internal root CA
  • To support split-horizon DNS and NAT scenarios
  • To ensure a similar experience to standalone control planes, where you can use functions, such as the Show Login Command function, with the correct kubeconfig and DNS configuration

You can define a DNS name either during your initial setup or during postinstallation operations, by entering a domain name in the kubeAPIServerDNSName parameter of a HostedCluster object.

Prerequisites

  • You have a valid TLS certificate that covers the DNS name that you set in the kubeAPIServerDNSName parameter.
  • You have a resolvable DNS name URI that can reach and point to the correct address.

Procedure

  • In the specification for the HostedCluster object, add the kubeAPIServerDNSName parameter and the address for the domain and specify which certificate to use, as shown in the following example:

    #...
spec:
  configuration:
    apiServer:
      servingCerts:
        namedCertificates:
        - names:
          - xxx.example.com
          - yyy.example.com
          servingCertificate:
            name: <my_serving_certificate>
  kubeAPIServerDNSName: <custom_address>

    The value for the kubeAPIServerDNSName parameter must be a valid and addressable domain.

    After you define the kubeAPIServerDNSName parameter and specify the certificate, the Control Plane Operator controllers create a kubeconfig file named custom-admin-kubeconfig, where the file gets stored in the HostedControlPlane namespace. The generation of certificates happen from the root CA, and the HostedControlPlane namespace manages their expiration and renewal.

    The Control Plane Operator reports a new kubeconfig file named CustomKubeconfig in the HostedControlPlane namespace. That file uses the defined new server in the kubeAPIServerDNSName parameter.

    A reference for the custom kubeconfig file exists in the status parameter as CustomKubeconfig of the HostedCluster object. The CustomKubeConfig parameter is optional, and you can add the parameter only if the kubeAPIServerDNSName parameter is not empty. After you set the CustomKubeConfig parameter, the parameter triggers the generation of a secret named <hosted_cluster_name>-custom-admin-kubeconfig in the HostedCluster namespace. You can use the secret to access the HostedCluster API server. If you remove the CustomKubeConfig parameter during postinstallation operations, deletion of all related secrets and status references occur.

    Note

    Defining a custom DNS name does not directly impact the data plane, so no expected rollouts occur. The HostedControlPlane namespace receives the changes from the HyperShift Operator and deletes the corresponding parameters.

    If you remove the kubeAPIServerDNSName parameter from the specification for the HostedCluster object, all newly generated secrets and the CustomKubeconfig reference are removed from the cluster and from the status parameter.

Customized ingress and DNS behavior

If you do not want to use the default ingress and DNS behavior, you can configure a KubeVirt hosted cluster with a unique base domain at creation time.

This option requires manual configuration steps during creation and involves three main steps: cluster creation, load balancer creation, and wildcard DNS configuration.

Creating a hosted cluster that specifies the base domain

If you do not want to use the default ingress and DNS behavior, you can configure a KubeVirt hosted cluster with a unique base domain at creation time.

Procedure

  1. Create the cluster by entering the following command:

    $ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \
  --node-pool-replicas <worker_count> \
  --pull-secret <path_to_pull_secret> \
  --memory <value_for_memory> \
  --cores <value_for_cpu> \
  --base-domain <basedomain> \
  --arch <architecture_of_the_nodepool> \
  --release-image <ocp_release_image_for_the_cluster> \
  --image-content-sources <path_to_image_content_sources_file> \
  --additional-trust-bundle <path_to_ca_bundle_file>

    • --name specifies the name of your hosted cluster.

    • --node-pool-replicas specifies the worker count, for example, 2.

    • --pull-secret specifies the path to your pull secret, for example, /user/name/pullsecret.

    • --memory specifies a value for memory, for example, 6Gi.

    • --cores specifies a value for CPU, for example, 2.

    • --base-domain specifies the base domain, for example, hypershift.lab.

    • --arch specifies the architecture of the node pool, for example, s390x. The default is amd64.

    • --release-image specifies the ocp release image for the cluster, for example, quay.io/openshift-release-dev/ocp-release:4.20.14-multi.

    • --image-content-sources specifies the path to a file with image content sources.

    • --additional-trust-bundle specifies the path to a file with user CA bundle.

      As a result, the hosted cluster has an ingress wildcard that is configured for the cluster name and the base domain, for example, .apps.example.hypershift.lab. The hosted cluster remains in Partial status because after you create a hosted cluster with unique base domain, you must configure the required DNS records and load balancer.

Verification

  1. View the status of your hosted cluster by entering the following command:

    $ oc get --namespace clusters hostedclusters
    Example output
    NAME            VERSION   KUBECONFIG                       PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
example                   example-admin-kubeconfig         Partial    True        False         The hosted control plane is available

  2. Access the cluster by entering the following commands:

    $ hcp create kubeconfig --name <hosted_cluster_name> \
  > <hosted_cluster_name>-kubeconfig

    $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get co
    Example output
    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    <4.x.0>     False       False         False      30m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.example.hypershift.lab): Get "https://console-openshift-console.apps.example.hypershift.lab": dial tcp: lookup console-openshift-console.apps.example.hypershift.lab on 172.31.0.10:53: no such host
ingress                                    <4.x.0>     True        False         True       28m     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

    Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use.

  3. To fix any errors in the output, complete the steps in "Setting up the load balancer" and "Setting up a wildcard DNS".

    Note

    If your hosted cluster is on bare metal, you might need MetalLB to set up load balancer services. For more information, see "Configuring MetalLB".

Setting up the load balancer

Set up the load balancer service that routes ingress traffic to the KubeVirt VMs and assigns a wildcard DNS entry to the load balancer IP address.

When you create a LoadBalancer service on an OpenShift Virtualization hosted cluster, a mirror LoadBalancer service is automatically created and configured on the management cluster to expose the service. As a result, the hosted cluster LoadBalancer services are exposed through the same technology that the management cluster has configured for its own LoadBalancer services, such as MetalLB.

Procedure

  1. A NodePort service that exposes the hosted cluster ingress already exists. You can export the node ports and create the load balancer service that targets those ports.

    1. Get the HTTP node port by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get services \
  -n openshift-ingress router-nodeport-default \
  -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}'

      Note the HTTP node port value to use in the next step.

    2. Get the HTTPS node port by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get services \
  -n openshift-ingress router-nodeport-default \
  -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}'

      Note the HTTPS node port value to use in the next step.

  2. Enter the following information in a YAML file:

    apiVersion: v1
kind: Service
metadata:
  labels:
    app: <hosted_cluster_name>
  name: <hosted_cluster_name>-apps
  namespace: clusters-<hosted_cluster_name>
spec:
  ports:
  - name: https-443
    port: 443
    protocol: TCP
    targetPort: <https_node_port>
  - name: http-80
    port: 80
    protocol: TCP
    targetPort: <http_node_port>
  selector:
    kubevirt.io: virt-launcher
  type: LoadBalancer

    where:

    • <https_node_port> specifies the HTTPS node port value that you noted in the previous step.
    • <http_node_port> specifies the HTTP node port value that you noted in the previous step.

  3. Create the load balancer service by running the following command:

    $ oc create -f <file_name>.yaml

Setting up a wildcard DNS

If you are customizing the ingress and DNS for your hosted cluster, you need to set up a wildcard DNS record or CNAME that references the external IP of the load balancer service.

Procedure

  1. Get the external IP address by entering the following command:

    $ oc -n clusters-<hosted_cluster_name> get service <hosted-cluster-name>-apps \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
    Example output
    192.168.20.30

  2. Configure a wildcard DNS entry that references the external IP address. View the following example DNS entry:

    *.apps.<hosted_cluster_name\>.<base_domain\>.

    The DNS entry must be able to route inside and outside of the cluster.

    DNS resolutions example
    dig +short test.apps.example.hypershift.lab

192.168.20.30

Verification

  • Check that hosted cluster status has moved from Partial to Completed by entering the following command:

    $ oc get --namespace clusters hostedclusters
    Example output
    NAME            VERSION   KUBECONFIG                       PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
example         <4.x.0>     example-admin-kubeconfig         Completed   True        False         The hosted control plane is available

    Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use.

Additional networks, guaranteed CPUs, and VM scheduling for node pools

You can configure additional networks for node pools, request a guaranteed CPU access for Virtual Machines (VMs), or manage scheduling of KubeVirt VMs.

Adding multiple networks to a node pool

By default, nodes generated by a node pool are attached to the pod network. You can attach additional networks to the nodes by using NetworkAttachmentDefinitions.

Warning

You can connect compute nodes to multiple networks that have a Linux bridge or a user-defined network (UDN) that uses the localnet topology. Red Hat does not support connecting compute nodes to multiple UDNs that use a layer 2 or a layer 3 overlay topology.

Procedure

  • To add multiple networks to nodes, use the --additional-network argument by running the following command:

    $ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \
  --node-pool-replicas <worker_node_count> \
  --pull-secret <path_to_pull_secret> \
  --memory <memory> \
  --cores <cpu> \
  --additional-network name:<namespace/name> \
  –-additional-network name:<namespace/name>

    where:

    hosted_cluster_name
    Specifies the name of your hosted cluster, for example, my-hosted-cluster.
worker_node_count
Specifies your compute node count, for example, 2.
path_to_pull_secret
Specifies the path to your pull secret, for example, /user/name/pullsecret.
memory
Specifies the memory value, for example, 8Gi.
cpu
Specifies the CPU value, for example, 2.
namespace/name
Specifies the value of the --additional-network argument to name:<namespace/name>. Replace <namespace/name> with a namespace and name of your NetworkAttachmentDefinitions.

Using an additional network as default

You can add your additional network as a default network for the nodes by disabling the default pod network.

Procedure

  • To add an additional network as default to your nodes, run the following command:

    $ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \
  --node-pool-replicas <worker_node_count> \
  --pull-secret <path_to_pull_secret> \
  --memory <memory> \
  --cores <cpu> \
  --attach-default-network false \
  --additional-network name:<namespace>/<network_name>
    • --name specifies the name of your hosted cluster, for example, my-hosted-cluster.
    • --node-pool-replicas specifies your worker node count, for example, 2.
    • --pull-secret specifies the path to your pull secret, for example, /user/name/pullsecret.
    • --memory specifies the memory value, for example, 8Gi.
    • --cores specifies the CPU value, for example, 2.
    • --attach-default-network false disables the default pod network.
    • --additional-network specifies the additional network that you want to add to your nodes, for example, name:my-namespace/my-network.

Requesting guaranteed CPU resources

By default, KubeVirt VMs might share their CPUs with other workloads on a node. This might impact performance of a VM. To avoid the performance impact, you can request a guaranteed CPU access for VMs.

Procedure

  • To request guaranteed CPU resources, set the --qos-class argument to Guaranteed by running the following command:

    $ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \
  --node-pool-replicas <worker_node_count> \
  --pull-secret <path_to_pull_secret> \
  --memory <memory> \
  --cores <cpu> \
  --qos-class Guaranteed
    • --name specifies the name of your hosted cluster, for example, my-hosted-cluster.
    • --node-pool-replicas specifies your worker node count, for example, 2.
    • --pull-secret specifies the path to your pull secret, for example, /user/name/pullsecret.
    • --memory specifies the memory value, for example, 8Gi.
    • --cores specifies the CPU value, for example, 2.
    • --qos-class Guaranteed guarantees that the specified number of CPU resources are assigned to VMs.

Scheduling KubeVirt VMs on a set of nodes

By default, KubeVirt virtual machines (VMs) created by a node pool are scheduled to any available nodes. You can schedule KubeVirt VMs on a specific set of nodes that has enough capacity to run the VM.

Procedure

  • To schedule KubeVirt VMs within a node pool on a specific set of nodes, use the --vm-node-selector argument by running the following command:

    $ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \
  --node-pool-replicas <worker_node_count> \
  --pull-secret <path_to_pull_secret> \
  --memory <memory> \
  --cores <cpu> \
  --vm-node-selector <label_key>=<label_value>,<label_key>=<label_value>
    • --name specifies the name of your hosted cluster, for example, my-hosted-cluster.
    • --node-pool-replicas specifies your worker node count, for example, 2.
    • --pull-secret specifies the path to your pull secret, for example, /user/name/pullsecret.
    • --memory specifies the memory value, for example, 8Gi.
    • --cores specifies the CPU value, for example, 2.
    • --vm-node-selector defines a specific set of nodes that contains the key-value pairs. Replace <label_key> with the keys of your labels and replace <label_value> with the values of your labels.

Scaling a node pool

You can manually scale a node pool for a hosted cluster on OpenShift Virtualization by using the oc scale command.

Procedure

  1. Run the following command:

    NODEPOOL_NAME=${CLUSTER_NAME}-work
NODEPOOL_REPLICAS=5

$ oc scale nodepool/$NODEPOOL_NAME --namespace clusters \
  --replicas=$NODEPOOL_REPLICAS

  2. After a few moments, enter the following command to see the status of the node pool:

    $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes
    Example output
    NAME                  STATUS   ROLES    AGE     VERSION
example-9jvnf         Ready    worker   97s     v1.27.4+18eadca
example-n6prw         Ready    worker   116m    v1.27.4+18eadca
example-nc6g4         Ready    worker   117m    v1.27.4+18eadca
example-thp29         Ready    worker   4m17s   v1.27.4+18eadca
example-twxns         Ready    worker   88s     v1.27.4+18eadca

Additional resources

Adding node pools

You can create node pools for a hosted cluster by specifying a name, number of replicas, and any additional information, such as memory and CPU requirements.

Procedure

  1. If the management cluster has a cluster-wide proxy configured, you must configure proxy settings in the HostedCluster resource by completing the following steps:

    1. Edit the HostedCluster resource by entering the following command:

      $ oc edit hc <hosted_cluster_name> -n <hosted_cluster_namespace>

    2. In the HostedCluster resource, add the proxy configuration as shown in the following example:

      apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  annotations:
# ...
    hypershift.openshift.io/HasBeenAvailable: "true"
    hypershift.openshift.io/management-platform: VSphere
# ...
  name: <hosted_cluster_name>
  namespace: <hosted_cluster_namespace>
# ...
spec:
# ...
  clusterID: fa45babd-40f3-4085-9b30-8bc3b7df1557
  configuration:
    proxy:
      httpProxy: http://web-proxy.example.com:3128
      httpsProxy: http://web-proxy.example.com:3128
      noProxy: .example.com,192.168.10.0/24

      In the spec.configuration.proxy fields, specify the details of the proxy configuration.

    3. Check the status on the management cluster by entering the following command:

      $ oc get nodepool -n <hosted_cluster_namespace>

    4. Check the status on the hosted cluster by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get nodes

  2. To create a node pool, enter the following information. In this example, the node pool has more CPUs assigned to the VMs:

    export NODEPOOL_NAME=${CLUSTER_NAME}-extra-cpu
export WORKER_COUNT="2"
export MEM="6Gi"
export CPU="4"
export DISK="16"

$ hcp create nodepool kubevirt \
  --cluster-name $CLUSTER_NAME \
  --name $NODEPOOL_NAME \
  --node-count $WORKER_COUNT \
  --memory $MEM \
  --cores $CPU \
  --root-volume-size $DISK

  3. Check the status of the node pool by listing nodepool resources in the namespace:

    $ oc get nodepools --namespace <hosted_cluster_namespace>
    Example output
    NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
example                   example         5               5               False         False        <4.x.0>
example-extra-cpu         example         2                               False         False                  True              True             Minimum availability requires 2 replicas, current 0 available

    Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use.

Verification

  1. After some time, you can check the status of the node pool by entering the following command:

    $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes
    Example output
    NAME                      STATUS   ROLES    AGE     VERSION
example-9jvnf             Ready    worker   97s     v1.27.4+18eadca
example-n6prw             Ready    worker   116m    v1.27.4+18eadca
example-nc6g4             Ready    worker   117m    v1.27.4+18eadca
example-thp29             Ready    worker   4m17s   v1.27.4+18eadca
example-twxns             Ready    worker   88s     v1.27.4+18eadca
example-extra-cpu-zh9l5   Ready    worker   2m6s    v1.27.4+18eadca
example-extra-cpu-zr8mj   Ready    worker   102s    v1.27.4+18eadca

  2. Verify that the node pool is in the status that you expect by entering this command:

    $ oc get nodepools --namespace <hosted_cluster_namespace>
    Example output
    NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
example                   example         5               5               False         False        <4.x.0>
example-extra-cpu         example         2               2               False         False        <4.x.0>

    Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use.

Additional resources

Verifying hosted cluster creation on OpenShift Virtualization

After you create a hosted cluster on OpenShift Virtualization, verify that it was successfully created.

Procedure

  1. Verify that the HostedCluster resource transitioned to the completed state by entering the following command:

    $ oc get --namespace clusters hostedclusters <hosted_cluster_name>
    Example output
    NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
clusters    example   4.12.2    example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

  2. Verify that all the cluster operators in the hosted cluster are online by entering the following commands:

    $ hcp create kubeconfig --name <hosted_cluster_name> \
  > <hosted_cluster_name>-kubeconfig

    $ oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig
    Example output
    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    4.12.2   True        False         False      2m38s
csi-snapshot-controller                    4.12.2   True        False         False      4m3s
dns                                        4.12.2   True        False         False      2m52s
image-registry                             4.12.2   True        False         False      2m8s
ingress                                    4.12.2   True        False         False      22m
kube-apiserver                             4.12.2   True        False         False      23m
kube-controller-manager                    4.12.2   True        False         False      23m
kube-scheduler                             4.12.2   True        False         False      23m
kube-storage-version-migrator              4.12.2   True        False         False      4m52s
monitoring                                 4.12.2   True        False         False      69s
network                                    4.12.2   True        False         False      4m3s
node-tuning                                4.12.2   True        False         False      2m22s
openshift-apiserver                        4.12.2   True        False         False      23m
openshift-controller-manager               4.12.2   True        False         False      23m
openshift-samples                          4.12.2   True        False         False      2m15s
operator-lifecycle-manager                 4.12.2   True        False         False      22m
operator-lifecycle-manager-catalog         4.12.2   True        False         False      23m
operator-lifecycle-manager-packageserver   4.12.2   True        False         False      23m
service-ca                                 4.12.2   True        False         False      4m41s
storage                                    4.12.2   True        False         False      4m43s