--- title: Deploying hosted control planes on OpenShift Virtualization --- # Deploying hosted control planes on OpenShift Virtualization {#hcp-deploy-virt} With hosted control planes and OpenShift Virtualization, you can create OpenShift Container Platform clusters with worker nodes that are hosted by KubeVirt virtual machines. Hosted control planes on OpenShift Virtualization provides several benefits: - Enhances resource usage by packing hosted control planes and hosted clusters in the same underlying bare-metal infrastructure - Separates hosted control planes and hosted clusters to provide strong isolation - Reduces cluster provision time by eliminating the bare-metal node bootstrapping process - Manages many releases under the same base OpenShift Container Platform cluster The hosted control planes feature is enabled by default. You can use the hosted control plane command-line interface, `hcp`, to create an OpenShift Container Platform hosted cluster. The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine Operator". **Additional resources** {._additional-resources} - [Disabling the automatic import of hosted clusters into multicluster engine Operator](/openshift-docs-markdown/hosted_control_planes/hcp-import#hcp-import-disable_hcp-import) ## Prerequisites to deploy hosted control planes on OpenShift Virtualization {#hcp-virt-prereqs_hcp-deploy-virt} To create an OpenShift Container Platform cluster on OpenShift Virtualization, you must meet several prerequisites. - You have administrator access to an OpenShift Container Platform cluster, version 4.14 or later, specified in the `KUBECONFIG` environment variable. - The OpenShift Container Platform management cluster must have wildcard DNS routes enabled, as shown in the following command: ```terminal $ oc patch ingresscontroller -n openshift-ingress-operator default \ --type=json \ -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]' ``` - The OpenShift Container Platform management cluster has OpenShift Virtualization, version 4.14 or later, installed on it. For more information, see "Installing OpenShift Virtualization using the web console". - The OpenShift Container Platform management cluster is on-premise bare metal. - The OpenShift Container Platform management cluster must be configured with `OVNKubernetes` as the default pod network Container Network Interface (CNI). Live migration is supported for nodes only if the CNI is OVN-Kubernetes. - The OpenShift Container Platform management cluster has a default storage class. For more information, see "Postinstallation storage configuration". The following example shows how to set a default storage class: ```terminal $ oc patch storageclass ocs-storagecluster-ceph-rbd \ -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` - When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using Logical Volume Manager storage". - On the OpenShift Container Platform cluster that hosts the OpenShift Virtualization virtual machines, you must use a `ReadWriteMany` (RWX) storage class so that live migration can be enabled. - You have a valid pull secret file for the `quay.io/openshift-release-dev` repository. For more information, see "Install OpenShift on any x86_64 platform with user-provisioned infrastructure". - You have installed the hosted control plane command-line interface. - You have configured a load balancer. For more information, see "Configuring MetalLB". - For optimal network performance, you are using a network maximum transmission unit (MTU) of 9000 or greater on the OpenShift Container Platform cluster that hosts the KubeVirt virtual machines. If you use a lower MTU setting, network latency and the throughput of the hosted pods are affected. Enable multiqueue on node pools only when the MTU is 9000 or greater. > [!IMPORTANT] > You cannot change the MTU value for your cluster as a postinstallation task. - The multicluster engine Operator has at least one managed OpenShift Container Platform cluster. The `local-cluster` is automatically imported. For more information about the `local-cluster`, see "Advanced configuration" in the multicluster engine Operator documentation. You can check the status of your hub cluster by running the following command: ```terminal $ oc get managedclusters local-cluster ``` - Ensure each hosted cluster has a cluster-wide unique name. - Do not use `clusters` as a hosted cluster name. - Do not create a hosted cluster in the namespace of a multicluster engine Operator managed cluster. **Additional resources** {._additional-resources} - [Installing OpenShift Virtualization using the web console](/openshift-docs-markdown/virt/install/installing-virt#installing-virt-web) - [Postinstallation storage configuration](/openshift-docs-markdown/post_installation_configuration/post-install-storage-configuration#post-install-storage-configuration) - [Install OpenShift on any x86_64 platform with user-provisioned infrastructure](https://console.redhat.com/openshift/install/platform-agnostic/user-provisioned) - [Configuring MetalLB](/openshift-docs-markdown/hosted_control_planes/hcp-deploy/hcp-deploy-virt#hcp-metallb_hcp-deploy-virt) - [Advanced configuration (Red Hat Advanced Cluster Management documentation)](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/latest/html/clusters/cluster_mce_overview#advanced-config-engine) - [Recommended etcd practices](/openshift-docs-markdown/etcd/etcd-practices#recommended-etcd-practices) - [Persistent storage using Logical Volume Manager Storage](/openshift-docs-markdown/storage/persistent_storage_local/persistent-storage-using-lvms#persistent-storage-using-lvms) ### OpenShift Virtualization firewall and port requirements {#hcp-virt-firewall-port_hcp-deploy-virt} Ensure that you meet the firewall and port requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters. - The `kube-apiserver` service runs on port 6443 by default and requires ingress access for communication between the control plane components. - If you use the `NodePort` publishing strategy, ensure that the node port that is assigned to the `kube-apiserver` service is exposed. - If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses. - If you use the `NodePort` publishing strategy, use a firewall rule for the `ignition-server` and `Oauth-server` settings. - The `konnectivity` agent, which establishes a reverse tunnel to allow bi-directional communication on the hosted cluster, requires egress access to the cluster API server address on port 6443. With that egress access, the agent can reach the `kube-apiserver` service. - If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443. - If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes. - If you change the default port of 6443, adjust the rules to reflect that change. - Ensure that you open any ports that are required by the workloads that run in the clusters. - Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. - For production deployments, use a load balancer to simplify access through a single IP address. ## Live migration for compute nodes {#hcp-virt-live-migration_hcp-deploy-virt} While the management cluster for hosted cluster virtual machines (VMs) is undergoing updates or maintenance, the hosted cluster VMs can be automatically live migrated to prevent disrupting hosted cluster workloads. As a result, the management cluster can be updated without affecting the availability and operation of the KubeVirt platform hosted clusters. > [!IMPORTANT] > The live migration of KubeVirt VMs is enabled by default provided that the VMs use `ReadWriteMany` (RWX) storage for both the root volume and the storage classes that are mapped to the `kubevirt-csi` CSI provider. You can verify that the VMs in a node pool are capable of live migration by checking the `KubeVirtNodesLiveMigratable` condition in the `status` section of a `NodePool` object. In the following example, the VMs cannot be live migrated because RWX storage is not used. ```yaml {title="Example configuration where VMs cannot be live migrated"} - lastTransitionTime: "2024-10-08T15:38:19Z" message: | 3 of 3 machines are not live migratable Machine user-np-ngst4-gw2hz: DisksNotLiveMigratable: user-np-ngst4-gw2hz is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-gw2hz-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) Machine user-np-ngst4-npq7x: DisksNotLiveMigratable: user-np-ngst4-npq7x is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-npq7x-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) Machine user-np-ngst4-q5nkb: DisksNotLiveMigratable: user-np-ngst4-q5nkb is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-q5nkb-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) observedGeneration: 1 reason: DisksNotLiveMigratable status: "False" type: KubeVirtNodesLiveMigratable ``` In the next example, the VMs meet the requirements to be live migrated. ```yaml {title="Example configuration where VMs can be live migrated"} - lastTransitionTime: "2024-10-08T15:38:19Z" message: "All is well" observedGeneration: 1 reason: AsExpected status: "True" type: KubeVirtNodesLiveMigratable ``` While live migration can protect VMs from disruption in normal circumstances, events such as infrastructure node failure can result in a hard restart of any VMs that are hosted on the failed node. For live migration to be successful, the source node that a VM is hosted on must be working correctly. When the VMs in a node pool cannot be live migrated, workload disruption might occur on the hosted cluster during maintenance on the management cluster. By default, the hosted control planes controllers try to drain the workloads that are hosted on KubeVirt VMs that cannot be live migrated before the VMs are stopped. Draining the hosted cluster nodes before stopping the VMs allows pod disruption budgets to protect workload availability within the hosted cluster. ## Configuring MetalLB {#hcp-metallb_hcp-deploy-virt} Before you can create a hosted cluster on the KubeVirt platform, you must have the MetalLB load balancer configured. **Prerequisites** - You have installed the MetalLB Operator. For more information, see "Installing the MetalLB Operator". **Procedure** 1. Create a `MetalLB` resource by saving the following sample YAML content in the `configure-metallb.yaml` file: ```yaml apiVersion: metallb.io/v1beta1 kind: MetalLB metadata: name: metallb namespace: metallb-system ``` 2. Apply the YAML content by entering the following command: ```terminal $ oc apply -f configure-metallb.yaml ``` ```terminal {title="Example output"} metallb.metallb.io/metallb created ``` 3. Create a `IPAddressPool` resource by saving the following sample YAML content in the `create-ip-address-pool.yaml` file: ```yaml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: metallb namespace: metallb-system spec: addresses: - 192.168.216.32-192.168.216.122 ``` Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network. 4. Apply the YAML content by entering the following command: ```terminal $ oc apply -f create-ip-address-pool.yaml ``` ```terminal {title="Example output"} ipaddresspool.metallb.io/metallb created ``` 5. Create a `L2Advertisement` resource by saving the following sample YAML content in the `l2advertisement.yaml` file: ```yaml apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: l2advertisement namespace: metallb-system spec: ipAddressPools: - metallb ``` 6. Apply the YAML content by entering the following command: ```terminal $ oc apply -f l2advertisement.yaml ``` ```terminal {title="Example output"} l2advertisement.metallb.io/metallb created ``` **Additional resources** {._additional-resources} - [Installing the MetalLB Operator](/openshift-docs-markdown/networking/networking_operators/metallb-operator/metallb-operator-install#metallb-operator-install_metallb-operator-install) ## Hosted clusters with the KubeVirt platform {#hcp-virt-create-hc_hcp-deploy-virt} With OpenShift Container Platform 4.14 or later, you can create a hosted cluster with KubeVirt by using the command-line interface (CLI), the console, or by using external infrastructure. ### Creating a hosted cluster with the KubeVirt platform by using the CLI {#hcp-virt-create-hc-cli_hcp-deploy-virt} To create a hosted cluster on OpenShift Virtualization, you can use the hosted control plane command-line interface (CLI), `hcp`. > [!IMPORTANT] > Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels. **Procedure** 1. Create a hosted cluster with the KubeVirt platform by entering the following command: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --etcd-storage-class= \ --arch \ --release-image \ --image-content-sources \ --disable-cluster-capabilities= \ --enable-cluster-capabilities= \ --additional-trust-bundle ``` - `--name` defines the name of your hosted cluster, for example, `my-hosted-cluster`. - `--node-pool-replicas` defines the node pool replica count, for example, `3`. You must specify the replica count as `0` or greater to create the same number of replicas. Otherwise, no node pools are created. - `--pull-secret` defines the path to your pull secret, for example, `/user/name/pullsecret`. - `--memory` defines a value for memory, for example, `6Gi`. - `--cores` defines a value for CPU, for example, `2`. - `--etcd-storage-class` defines the etcd storage class name, for example, `lvm-storageclass`. - `--arch` defines the architecture of the node pool, for example, `s390x`. The default is `amd64`. - `--release-image` defines the OpenShift Container Platform release image for the cluster, for example, `quay.io/openshift-release-dev/ocp-release:4.20.14-multi`. You can use the `--release-image` flag to set up the hosted cluster with a specific OpenShift Container Platform release. - `--image-content-sources` specifies the path to a file with image content sources. - `--disable-cluster-capabilities` specifies that you want to disable optional capabilities. This flag is optional. For more information, see "Capabilities for hosted clusters". - `--enable-cluster-capabilities` specifies that you want to enable optional capabilities in the hosted cluster. This flag is optional. For more information, see "Capabilities for hosted clusters". - `--additional-trust-bundle` specifies the path to a file with user CA bundle. A default node pool is created for the cluster with a specific number of virtual machine worker replicas according to the `--node-pool-replicas` flag. 2. After a few moments, verify that the hosted control plane pods are running by entering the following command: ```terminal $ oc -n clusters- get pods ``` ```terminal {title="Example output"} NAME READY STATUS RESTARTS AGE capi-provider-5cc7b74f47-n5gkr 1/1 Running 0 3m catalog-operator-5f799567b7-fd6jw 2/2 Running 0 69s certified-operators-catalog-784b9899f9-mrp6p 1/1 Running 0 66s cluster-api-6bbc867966-l4dwl 1/1 Running 0 66s . . . redhat-operators-catalog-9d5fd4d44-z8qqk 1/1 Running 0 66s ``` A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned. **Verification** - To check the status of the hosted cluster, see the corresponding `HostedCluster` resource by entering the following command: ```terminal $ oc get --namespace clusters hostedclusters ``` See the following example output, which illustrates a fully provisioned `HostedCluster` object: ``` NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE clusters my-hosted-cluster <4.x.0> example-admin-kubeconfig Completed True False The hosted control plane is available ``` Replace `<4.x.0>` with the supported OpenShift Container Platform version that you want to use. **Additional resources** {._additional-resources} - [Labeling management cluster nodes](/openshift-docs-markdown/hosted_control_planes/hcp-prepare/hcp-distribute-workloads#hcp-labels-taints_hcp-distribute-workloads) - [Configuring a custom API server certificate in a hosted cluster](/openshift-docs-markdown/hosted_control_planes/hcp-certificates#hcp-custom-cert_hcp-certificates) ### Creating a hosted cluster with the KubeVirt platform by using external infrastructure {#hcp-virt-create-hc-ext-infra_hcp-deploy-virt} By default, the HyperShift Operator hosts both the control plane pods of the hosted cluster and the KubeVirt worker VMs within the same cluster. With the external infrastructure feature, you can place the worker node VMs on a separate cluster from the control plane pods. - The *management cluster* is the OpenShift Container Platform cluster that runs the HyperShift Operator and hosts the control plane pods for a hosted cluster. - The *infrastructure cluster* is the OpenShift Container Platform cluster that runs the KubeVirt worker VMs for a hosted cluster. - By default, the management cluster also acts as the infrastructure cluster that hosts VMs. However, for external infrastructure, the management and infrastructure clusters are different. > [!IMPORTANT] > Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels. **Prerequisites** - You must have a namespace on the external infrastructure cluster for the KubeVirt nodes to be hosted in. - You must have a `kubeconfig` file for the external infrastructure cluster. **Procedure** - In the `hcp` command-line interface, place the KubeVirt worker VMs on the infrastructure cluster, use the `--infra-kubeconfig-file` and `--infra-namespace` arguments, as shown in the following example: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --infra-namespace=- \ --infra-kubeconfig-file= ``` - `--name` defines the name of your hosted cluster, for example, `my-hosted-cluster`. - `--node-pool-replicas` defines the worker count, for example, `2`. - `--pull-secret` defines the path to your pull secret, for example, `/user/name/pullsecret`. - `--memory` defines a value for memory, for example, `6Gi`. - `--cores` defines a value for CPU, for example, `2`. - `--infra-namespace` defines the infrastructure namespace, for example, `clusters-example`. - `--infra-kubeconfig-file` defines the path to your `kubeconfig` file for the infrastructure cluster, for example, `/user/name/external-infra-kubeconfig`. After you enter the command, the control plane pods are hosted on the management cluster that the HyperShift Operator runs on, and the KubeVirt VMs are hosted on a separate infrastructure cluster. **Additional resources** {._additional-resources} - [Labeling management cluster nodes](/openshift-docs-markdown/hosted_control_planes/hcp-prepare/hcp-distribute-workloads#hcp-labels-taints_hcp-distribute-workloads) - [Configuring a custom API server certificate in a hosted cluster](/openshift-docs-markdown/hosted_control_planes/hcp-certificates#hcp-custom-cert_hcp-certificates) ### Creating a hosted cluster by using the console {#hcp-virt-create-hc-console_hcp-deploy-virt} If you prefer to work in the OpenShift Container Platform console instead of the CLI, you can create a hosted cluster on the KubeVirt platform by using the console. > [!NOTE] > If you want to use predefined values to automatically populate fields in the console, you can create a OpenShift Virtualization credential. For more information, see "Creating a credential for an on-premises environment". **Procedure** 1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials. 2. In the console header, ensure that **All Clusters** is selected. 3. Click **Infrastructure > Clusters**. 4. Click **Create cluster > Red Hat OpenShift Virtualization > Hosted**. 5. On the **Create cluster** page, follow the prompts to enter details about the cluster and node pools. On the **Cluster details** page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a OpenShift Virtualization credential, the pull secret is automatically populated. > [!IMPORTANT] > Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels. 6. On the **Node pools** page, expand the **Networking options** section and configure the networking options for your node pool: 1. In the **Additional networks** field, enter a network name in the format of `/`; for example, `my-namespace/network1`. The namespace and the name must be valid DNS labels. Multiple networks are supported. 2. By default, the **Attach default pod network** checkbox is selected. You can clear this checkbox only if additional networks exist. 7. Review your entries and click **Create**. The **Hosted cluster** view is displayed. **Verification** 1. Monitor the deployment of the hosted cluster in the **Hosted cluster** view. If you do not see information about the hosted cluster, ensure that **All Clusters** is selected, and click the cluster name. 2. Wait until the control plane components are ready. This process can take a few minutes. 3. To view the node pool status, scroll to the **NodePool** section. The process to install the nodes takes about 10 minutes. You can also click **Nodes** to confirm whether the nodes joined the hosted cluster. **Additional resources** {._additional-resources} - [Labeling management cluster nodes](/openshift-docs-markdown/hosted_control_planes/hcp-prepare/hcp-distribute-workloads#hcp-labels-taints_hcp-distribute-workloads) - [Configuring a custom API server certificate in a hosted cluster](/openshift-docs-markdown/hosted_control_planes/hcp-certificates#hcp-custom-cert_hcp-certificates) - [Creating a credential for an on-premises environment (Red Hat Advanced Cluster Management documentation)](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.15/html/clusters/cluster_mce_overview#creating-a-credential-for-an-on-premises-environment) - [Accessing the hosted cluster](/openshift-docs-markdown/hosted_control_planes/hcp-manage/hcp-manage-virt#hcp-virt-access_hcp-manage-virt) ## Capabilities for hosted clusters {#hcp-cluster-capabilities_hcp-deploy-virt} To reduce resource consumption and prevent unnecessary Operators and operands from being deployed, administrators can enable or disable optional OpenShift Container Platform components when they create a hosted cluster. When capabilities are not specified on a `HostedCluster` resource, the cluster uses the OpenShift Container Platform version’s `DefaultCapabilitySet` settings, excluding the `baremetal` capability. As a result, most optional components are enabled by default. > [!IMPORTANT] > Capabilities are immutable after cluster creation. You cannot change them after you create the `HostedCluster` resource. ### Capabilities that you can enable or disable for a hosted cluster {#hcp-cluster-capabilities-ref_hcp-deploy-virt} Familiarize yourself with the supported capabilities that you can enable or disable for a `HostedCluster` resource. The capabilities are described in the following table: | Capability | Description | | --- | --- | | `ImageRegistry` | The OpenShift Image Registry Operator and its operands, including cloud storage infrastructure, such as S3 buckets and Identity and Access Management (IAM) users. | | `openshift-samples` | The OpenShift Samples Operator, which manages example `ImageStreams` and templates. | | `Insights` | The Insights Operator, which collects and uploads cluster telemetry data. | | `baremetal` | The Bare Metal Infrastructure Operator. This capability is excluded from the default set. If needed, you must explicitly enable it. | | `Console` | The OpenShift Web Console Operator and its operands. | | `NodeTuning` | The Node Tuning Operator, which manages node-level performance tuning by using TuneD and performance profiles. | | `Ingress` | The OpenShift Ingress Operator, which manages the default router of the cluster. | The following rules apply when you combine capability settings: No overlap : A capability cannot be in both the `enabled` and `disabled` lists simultaneously. Console requires Ingress : You can disable the `Ingress` capability only if the `Console` capability is also disabled because the console depends on Ingress. Version requirement : You must use OpenShift Container Platform 4.20 or later to disable any of the following capabilities: `openshift-samples`, `Insights`, `Console`, `NodeTuning`, and `Ingress`. You can disable `ImageRegistry` and `baremetal` on versions earlier than 4.20. Bare metal default exclusion : The `baremetal` capability is excluded from the default set. You can add it to the cluster by explicitly enabling it. ### Setting capabilities for a hosted cluster {#hcp-cluster-capabilities-proc_hcp-deploy-virt} To reduce unnecessary resource consumption, you can control which optional capabilities are enabled for a hosted cluster when you create the cluster. > [!IMPORTANT] > Capabilities are immutable after cluster creation. You cannot change them after you create the `HostedCluster` resource. You can specify which capabilities are enabled by either using the `hcp` command-line interface (CLI) or by setting the `HostedCluster` manifest. **Procedure** - To specify which capabilities are enabled in a hosted cluster by using the CLI, you can add the `--disable-cluster-capabilities` flag, the `--enable-cluster-capabilities` flag, or both. The following example shows how to disable the `ImageRegistry`, `Console`, and `Ingress` capabilities and enable the `baremetal` capability while you create a hosted cluster on AWS by using the `hcp` command-line interface: ```terminal $ hcp create cluster aws \ --name my-hosted-cluster \ --disable-cluster-capabilities=ImageRegistry,Console,Ingress \ --enable-cluster-capabilities=baremetal ``` You can specify multiple capabilities as a comma-separated list. The supported values are as follows: - `ImageRegistry` - `openshift-samples` - `Insights` - `baremetal` - `Console` - `NodeTuning` - `Ingress` - To specify which capabilities are enabled in a hosted cluster by using the `HostedCluster` manifest at cluster creation time, see the following examples: - To directly disable capabilities in a hosted cluster, add the `spec.capabilities.disabled` section in the `HostedCluster` resource: ```yaml apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster metadata: name: my-hosted-cluster namespace: my-cluster-namespace spec: capabilities: disabled: - ImageRegistry - Console - Ingress # ... ``` - To explicitly enable a capability that is not part of the default set of capabilities, such as the `baremetal` capability, see the following example: ```yaml apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster metadata: name: my-hosted-cluster namespace: my-cluster-namespace spec: capabilities: enabled: - baremetal # ... ``` - You can use both `enabled` and `disabled` if no capabilities are in both lists. See the following example: ```yaml apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster metadata: name: my-hosted-cluster namespace: my-cluster-namespace spec: capabilities: enabled: - baremetal disabled: - ImageRegistry - openshift-samples # ... ``` ## Configuring the default ingress and DNS for hosted control planes on OpenShift Virtualization {#hcp-virt-ingress-dns_hcp-deploy-virt} Every OpenShift Container Platform cluster includes a default application Ingress Controller, which must have an wildcard DNS record associated with it. By default, hosted clusters that are created by using OpenShift Virtualization automatically become a subdomain of the OpenShift Container Platform cluster that the virtual machines run on. For example, your OpenShift Container Platform cluster might have the following default ingress DNS entry: ```terminal *.apps.mgmt-cluster.example.com ``` As a result, a hosted cluster that is named `guest` and that runs on that underlying OpenShift Container Platform cluster has the following default ingress: ```terminal *.apps.guest.apps.mgmt-cluster.example.com ``` **Procedure** - For the default ingress DNS to work properly, the cluster that hosts the virtual machines must allow wildcard DNS routes. You can configure this behavior by entering the following command: ```terminal $ oc patch ingresscontroller -n openshift-ingress-operator default \ --type=json \ -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]' ``` > [!NOTE] > When you use the default hosted cluster ingress, connectivity is limited to HTTPS traffic over port 443. Plain HTTP traffic over port 80 is rejected. This limitation applies to only the default ingress behavior. ### Defining a custom DNS name {#hcp-custom-dns_hcp-deploy-virt} As a cluster administrator, you can create a hosted cluster with an external API DNS name that differs from the internal endpoint that gets used for node bootstraps and control plane communication. You might want to define a different DNS name for the following reasons: - To replace the user-facing TLS certificate with one from a public CA without breaking the control plane functions that bind to the internal root CA - To support split-horizon DNS and NAT scenarios - To ensure a similar experience to standalone control planes, where you can use functions, such as the `Show Login Command` function, with the correct `kubeconfig` and DNS configuration You can define a DNS name either during your initial setup or during postinstallation operations, by entering a domain name in the `kubeAPIServerDNSName` parameter of a `HostedCluster` object. **Prerequisites** - You have a valid TLS certificate that covers the DNS name that you set in the `kubeAPIServerDNSName` parameter. - You have a resolvable DNS name URI that can reach and point to the correct address. **Procedure** - In the specification for the `HostedCluster` object, add the `kubeAPIServerDNSName` parameter and the address for the domain and specify which certificate to use, as shown in the following example: ```yaml #... spec: configuration: apiServer: servingCerts: namedCertificates: - names: - xxx.example.com - yyy.example.com servingCertificate: name: kubeAPIServerDNSName: ``` The value for the `kubeAPIServerDNSName` parameter must be a valid and addressable domain. After you define the `kubeAPIServerDNSName` parameter and specify the certificate, the Control Plane Operator controllers create a `kubeconfig` file named `custom-admin-kubeconfig`, where the file gets stored in the `HostedControlPlane` namespace. The generation of certificates happen from the root CA, and the `HostedControlPlane` namespace manages their expiration and renewal. The Control Plane Operator reports a new `kubeconfig` file named `CustomKubeconfig` in the `HostedControlPlane` namespace. That file uses the defined new server in the `kubeAPIServerDNSName` parameter. A reference for the custom `kubeconfig` file exists in the `status` parameter as `CustomKubeconfig` of the `HostedCluster` object. The `CustomKubeConfig` parameter is optional, and you can add the parameter only if the `kubeAPIServerDNSName` parameter is not empty. After you set the `CustomKubeConfig` parameter, the parameter triggers the generation of a secret named `-custom-admin-kubeconfig` in the `HostedCluster` namespace. You can use the secret to access the `HostedCluster` API server. If you remove the `CustomKubeConfig` parameter during postinstallation operations, deletion of all related secrets and status references occur. > [!NOTE] > Defining a custom DNS name does not directly impact the data plane, so no expected rollouts occur. The `HostedControlPlane` namespace receives the changes from the HyperShift Operator and deletes the corresponding parameters. If you remove the `kubeAPIServerDNSName` parameter from the specification for the `HostedCluster` object, all newly generated secrets and the `CustomKubeconfig` reference are removed from the cluster and from the `status` parameter. ## Customized ingress and DNS behavior {#hcp-virt-ingress-dns-custom_hcp-deploy-virt} If you do not want to use the default ingress and DNS behavior, you can configure a KubeVirt hosted cluster with a unique base domain at creation time. This option requires manual configuration steps during creation and involves three main steps: cluster creation, load balancer creation, and wildcard DNS configuration. ### Creating a hosted cluster that specifies the base domain {#hcp-virt-hc-base-domain_hcp-deploy-virt} If you do not want to use the default ingress and DNS behavior, you can configure a KubeVirt hosted cluster with a unique base domain at creation time. **Procedure** 1. Create the cluster by entering the following command: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --base-domain \ --arch \ --release-image \ --image-content-sources \ --additional-trust-bundle ``` - `--name` specifies the name of your hosted cluster. - `--node-pool-replicas` specifies the worker count, for example, `2`. - `--pull-secret` specifies the path to your pull secret, for example, `/user/name/pullsecret`. - `--memory` specifies a value for memory, for example, `6Gi`. - `--cores` specifies a value for CPU, for example, `2`. - `--base-domain` specifies the base domain, for example, `hypershift.lab`. - `--arch` specifies the architecture of the node pool, for example, `s390x`. The default is `amd64`. - `--release-image` specifies the ocp release image for the cluster, for example, `quay.io/openshift-release-dev/ocp-release:4.20.14-multi`. - `--image-content-sources` specifies the path to a file with image content sources. - `--additional-trust-bundle` specifies the path to a file with user CA bundle. As a result, the hosted cluster has an ingress wildcard that is configured for the cluster name and the base domain, for example, `.apps.example.hypershift.lab`. The hosted cluster remains in `Partial` status because after you create a hosted cluster with unique base domain, you must configure the required DNS records and load balancer. **Verification** 1. View the status of your hosted cluster by entering the following command: ```terminal $ oc get --namespace clusters hostedclusters ``` ```terminal {title="Example output"} NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE example example-admin-kubeconfig Partial True False The hosted control plane is available ``` 2. Access the cluster by entering the following commands: ```terminal $ hcp create kubeconfig --name \ > -kubeconfig ``` ```terminal $ oc --kubeconfig -kubeconfig get co ``` ```terminal {title="Example output"} NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console <4.x.0> False False False 30m RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.example.hypershift.lab): Get "https://console-openshift-console.apps.example.hypershift.lab": dial tcp: lookup console-openshift-console.apps.example.hypershift.lab on 172.31.0.10:53: no such host ingress <4.x.0> True False True 28m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing) ``` Replace `<4.x.0>` with the supported OpenShift Container Platform version that you want to use. 3. To fix any errors in the output, complete the steps in "Setting up the load balancer" and "Setting up a wildcard DNS". > [!NOTE] > If your hosted cluster is on bare metal, you might need MetalLB to set up load balancer services. For more information, see "Configuring MetalLB". ### Setting up the load balancer {#hcp-virt-load-balancer_hcp-deploy-virt} Set up the load balancer service that routes ingress traffic to the KubeVirt VMs and assigns a wildcard DNS entry to the load balancer IP address. **Procedure** 1. A `NodePort` service that exposes the hosted cluster ingress already exists. You can export the node ports and create the load balancer service that targets those ports. 1. Get the HTTP node port by entering the following command: ```terminal $ oc --kubeconfig -kubeconfig get services \ -n openshift-ingress router-nodeport-default \ -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}' ``` Note the HTTP node port value to use in the next step. 2. Get the HTTPS node port by entering the following command: ```terminal $ oc --kubeconfig -kubeconfig get services \ -n openshift-ingress router-nodeport-default \ -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}' ``` Note the HTTPS node port value to use in the next step. 2. Enter the following information in a YAML file: ```yaml apiVersion: v1 kind: Service metadata: labels: app: name: -apps namespace: clusters- spec: ports: - name: https-443 port: 443 protocol: TCP targetPort: - name: http-80 port: 80 protocol: TCP targetPort: selector: kubevirt.io: virt-launcher type: LoadBalancer ``` where: - `` specifies the HTTPS node port value that you noted in the previous step. - `` specifies the HTTP node port value that you noted in the previous step. 3. Create the load balancer service by running the following command: ```terminal $ oc create -f .yaml ``` ### Setting up a wildcard DNS {#hcp-virt-wildcard-dns_hcp-deploy-virt} If you are customizing the ingress and DNS for your hosted cluster, you need to set up a wildcard DNS record or CNAME that references the external IP of the load balancer service. **Procedure** 1. Get the external IP address by entering the following command: ```terminal $ oc -n clusters- get service -apps \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` ```terminal {title="Example output"} 192.168.20.30 ``` 2. Configure a wildcard DNS entry that references the external IP address. View the following example DNS entry: ```terminal *.apps... ``` The DNS entry must be able to route inside and outside of the cluster. ```terminal {title="DNS resolutions example"} dig +short test.apps.example.hypershift.lab 192.168.20.30 ``` **Verification** - Check that hosted cluster status has moved from `Partial` to `Completed` by entering the following command: ```terminal $ oc get --namespace clusters hostedclusters ``` ```terminal {title="Example output"} NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE example <4.x.0> example-admin-kubeconfig Completed True False The hosted control plane is available ``` Replace `<4.x.0>` with the supported OpenShift Container Platform version that you want to use. ## Additional networks, guaranteed CPUs, and VM scheduling for node pools {#hcp-virt-addl_config_hcp-deploy-virt} You can configure additional networks for node pools, request a guaranteed CPU access for Virtual Machines (VMs), or manage scheduling of KubeVirt VMs. ### Adding multiple networks to a node pool {#hcp-virt-add-networks_hcp-deploy-virt} By default, nodes generated by a node pool are attached to the pod network. You can attach additional networks to the nodes by using `NetworkAttachmentDefinitions`. > [!IMPORTANT] > You can connect compute nodes to multiple networks that have a Linux bridge or a user-defined network (UDN) that uses the localnet topology. Red Hat does not support connecting compute nodes to multiple UDNs that use a layer 2 or a layer 3 overlay topology. **Procedure** - To add multiple networks to nodes, use the `--additional-network` argument by running the following command: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --additional-network name: \ –-additional-network name: ``` where: `hosted_cluster_name` : Specifies the name of your hosted cluster, for example, `my-hosted-cluster`. `worker_node_count` : Specifies your compute node count, for example, `2`. `path_to_pull_secret` : Specifies the path to your pull secret, for example, `/user/name/pullsecret`. `memory` : Specifies the memory value, for example, `8Gi`. `cpu` : Specifies the CPU value, for example, `2`. `namespace/name` : Specifies the value of the `--additional-network` argument to `name:`. Replace `` with a namespace and name of your `NetworkAttachmentDefinitions`. #### Using an additional network as default {#hcp-virt-addl-network_hcp-deploy-virt} You can add your additional network as a default network for the nodes by disabling the default pod network. **Procedure** - To add an additional network as default to your nodes, run the following command: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --attach-default-network false \ --additional-network name:/ ``` - `--name` specifies the name of your hosted cluster, for example, `my-hosted-cluster`. - `--node-pool-replicas` specifies your worker node count, for example, `2`. - `--pull-secret` specifies the path to your pull secret, for example, `/user/name/pullsecret`. - `--memory` specifies the memory value, for example, `8Gi`. - `--cores` specifies the CPU value, for example, `2`. - `--attach-default-network false` disables the default pod network. - `--additional-network` specifies the additional network that you want to add to your nodes, for example, `name:my-namespace/my-network`. ### Requesting guaranteed CPU resources {#hcp-virt-guaranteed-cpus_hcp-deploy-virt} By default, KubeVirt VMs might share their CPUs with other workloads on a node. This might impact performance of a VM. To avoid the performance impact, you can request a guaranteed CPU access for VMs. **Procedure** - To request guaranteed CPU resources, set the `--qos-class` argument to `Guaranteed` by running the following command: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --qos-class Guaranteed ``` - `--name` specifies the name of your hosted cluster, for example, `my-hosted-cluster`. - `--node-pool-replicas` specifies your worker node count, for example, `2`. - `--pull-secret` specifies the path to your pull secret, for example, `/user/name/pullsecret`. - `--memory` specifies the memory value, for example, `8Gi`. - `--cores` specifies the CPU value, for example, `2`. - `--qos-class Guaranteed` guarantees that the specified number of CPU resources are assigned to VMs. ### Scheduling KubeVirt VMs on a set of nodes {#hcp-virt-sched-vms_hcp-deploy-virt} By default, KubeVirt virtual machines (VMs) created by a node pool are scheduled to any available nodes. You can schedule KubeVirt VMs on a specific set of nodes that has enough capacity to run the VM. **Procedure** - To schedule KubeVirt VMs within a node pool on a specific set of nodes, use the `--vm-node-selector` argument by running the following command: ```terminal $ hcp create cluster kubevirt \ --name \ --node-pool-replicas \ --pull-secret \ --memory \ --cores \ --vm-node-selector =,= ``` - `--name` specifies the name of your hosted cluster, for example, `my-hosted-cluster`. - `--node-pool-replicas` specifies your worker node count, for example, `2`. - `--pull-secret` specifies the path to your pull secret, for example, `/user/name/pullsecret`. - `--memory` specifies the memory value, for example, `8Gi`. - `--cores` specifies the CPU value, for example, `2`. - `--vm-node-selector` defines a specific set of nodes that contains the key-value pairs. Replace `` with the keys of your labels and replace `` with the values of your labels. ## Scaling a node pool {#hcp-virt-scale-nodpool_hcp-deploy-virt} You can manually scale a node pool for a hosted cluster on OpenShift Virtualization by using the `oc scale` command. **Procedure** 1. Run the following command: ```terminal NODEPOOL_NAME=${CLUSTER_NAME}-work NODEPOOL_REPLICAS=5 $ oc scale nodepool/$NODEPOOL_NAME --namespace clusters \ --replicas=$NODEPOOL_REPLICAS ``` 2. After a few moments, enter the following command to see the status of the node pool: ```terminal $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION example-9jvnf Ready worker 97s v1.27.4+18eadca example-n6prw Ready worker 116m v1.27.4+18eadca example-nc6g4 Ready worker 117m v1.27.4+18eadca example-thp29 Ready worker 4m17s v1.27.4+18eadca example-twxns Ready worker 88s v1.27.4+18eadca ``` **Additional resources** {._additional-resources} - [Scaling up and down workloads in a hosted cluster](/openshift-docs-markdown/hosted_control_planes/hcp-machine-config#scale-up-down-autoscaler-hcp_hcp-machine-config) ### Adding node pools {#hcp-virt-add-node_hcp-deploy-virt} You can create node pools for a hosted cluster by specifying a name, number of replicas, and any additional information, such as memory and CPU requirements. **Procedure** 1. If the management cluster has a cluster-wide proxy configured, you must configure proxy settings in the `HostedCluster` resource by completing the following steps: 1. Edit the `HostedCluster` resource by entering the following command: ```terminal $ oc edit hc -n ``` 2. In the `HostedCluster` resource, add the proxy configuration as shown in the following example: ```yaml apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster metadata: annotations: # ... hypershift.openshift.io/HasBeenAvailable: "true" hypershift.openshift.io/management-platform: VSphere # ... name: namespace: # ... spec: # ... clusterID: fa45babd-40f3-4085-9b30-8bc3b7df1557 configuration: proxy: httpProxy: http://web-proxy.example.com:3128 httpsProxy: http://web-proxy.example.com:3128 noProxy: .example.com,192.168.10.0/24 ``` In the `spec.configuration.proxy` fields, specify the details of the proxy configuration. 3. Check the status on the management cluster by entering the following command: ```terminal $ oc get nodepool -n ``` 4. Check the status on the hosted cluster by entering the following command: ```terminal $ oc --kubeconfig -kubeconfig get nodes ``` 2. To create a node pool, enter the following information. In this example, the node pool has more CPUs assigned to the VMs: ```terminal export NODEPOOL_NAME=${CLUSTER_NAME}-extra-cpu export WORKER_COUNT="2" export MEM="6Gi" export CPU="4" export DISK="16" $ hcp create nodepool kubevirt \ --cluster-name $CLUSTER_NAME \ --name $NODEPOOL_NAME \ --node-count $WORKER_COUNT \ --memory $MEM \ --cores $CPU \ --root-volume-size $DISK ``` 3. Check the status of the node pool by listing `nodepool` resources in the namespace: ```terminal $ oc get nodepools --namespace ``` ```terminal {title="Example output"} NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE example example 5 5 False False <4.x.0> example-extra-cpu example 2 False False True True Minimum availability requires 2 replicas, current 0 available ``` Replace `<4.x.0>` with the supported OpenShift Container Platform version that you want to use. **Verification** 1. After some time, you can check the status of the node pool by entering the following command: ```terminal $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION example-9jvnf Ready worker 97s v1.27.4+18eadca example-n6prw Ready worker 116m v1.27.4+18eadca example-nc6g4 Ready worker 117m v1.27.4+18eadca example-thp29 Ready worker 4m17s v1.27.4+18eadca example-twxns Ready worker 88s v1.27.4+18eadca example-extra-cpu-zh9l5 Ready worker 2m6s v1.27.4+18eadca example-extra-cpu-zr8mj Ready worker 102s v1.27.4+18eadca ``` 2. Verify that the node pool is in the status that you expect by entering this command: ```terminal $ oc get nodepools --namespace ``` ```terminal {title="Example output"} NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE example example 5 5 False False <4.x.0> example-extra-cpu example 2 2 False False <4.x.0> ``` Replace `<4.x.0>` with the supported OpenShift Container Platform version that you want to use. **Additional resources** {._additional-resources} - [Scaling down the data plane to zero](/openshift-docs-markdown/hosted_control_planes/hcp-machine-config#scale-down-data-plane_hcp-machine-config) ## Verifying hosted cluster creation on OpenShift Virtualization {#hcp-virt-verify-hc_hcp-deploy-virt} After you create a hosted cluster on OpenShift Virtualization, verify that it was successfully created. **Procedure** 1. Verify that the `HostedCluster` resource transitioned to the `completed` state by entering the following command: ```terminal $ oc get --namespace clusters hostedclusters ``` ```terminal {title="Example output"} NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE clusters example 4.12.2 example-admin-kubeconfig Completed True False The hosted control plane is available ``` 2. Verify that all the cluster operators in the hosted cluster are online by entering the following commands: ```terminal $ hcp create kubeconfig --name \ > -kubeconfig ``` ```terminal $ oc get co --kubeconfig=-kubeconfig ``` ```terminal {title="Example output"} NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.12.2 True False False 2m38s csi-snapshot-controller 4.12.2 True False False 4m3s dns 4.12.2 True False False 2m52s image-registry 4.12.2 True False False 2m8s ingress 4.12.2 True False False 22m kube-apiserver 4.12.2 True False False 23m kube-controller-manager 4.12.2 True False False 23m kube-scheduler 4.12.2 True False False 23m kube-storage-version-migrator 4.12.2 True False False 4m52s monitoring 4.12.2 True False False 69s network 4.12.2 True False False 4m3s node-tuning 4.12.2 True False False 2m22s openshift-apiserver 4.12.2 True False False 23m openshift-controller-manager 4.12.2 True False False 23m openshift-samples 4.12.2 True False False 2m15s operator-lifecycle-manager 4.12.2 True False False 22m operator-lifecycle-manager-catalog 4.12.2 True False False 23m operator-lifecycle-manager-packageserver 4.12.2 True False False 23m service-ca 4.12.2 True False False 4m41s storage 4.12.2 True False False 4m43s ```