Deploying a virtualized control plane

After preparing your environment, install the virtualized control plane cluster by using your preferred installation method. The agent-based installer and GitOps Zero Touch Provisioning (ZTP) are the recommended methods for virtualized control plane deployments.

warning

KubeVirt Redfish is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Deploying a virtualized control plane with the Agent-based installer

Use the Agent-based Installer to deploy a virtualized control plane cluster. This method generates a bootable ISO containing all required configuration. You must mount the ISO to both the virtualized control plane nodes, by using KubeVirt Redfish, and the baremetal worker nodes, by using the Redfish API.

warning

When configuring the Agent-based Installer, the platform parameter in install-config.yaml determines the level of hardware management:

platform: baremetal

Enables full hardware awareness. The cluster installs the Bare Metal Operator, and uses it to manage node lifecycle, power states, and automatic remediation through the Redfish API.

platform: none

Treats infrastructure as generic compute. The cluster cannot control power states or boot orders. Requires external DNS and load balancing.

Prerequisites

• KubeVirt Redfish is installed and configured on the hosting cluster.
• Control plane VMs are created on the hosting cluster and powered off. In this state, you can manage the VM power lifecycle and boot configuration through KubeVirt Redfish.
• You have network access to the KubeVirt Redfish route.
• An HTTP server is available to host the agent ISO.

Procedure

1. Create an install-config.yaml file with BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: v1
   metadata:
     name: my-vcp-cluster
   baseDomain: example.com
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 3
   compute:
     - name: worker
       architecture: amd64
       hyperthreading: Enabled
       replicas: 2
   networking:
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
   platform:
     baremetal:
       provisioningNetwork: Disabled
       apiVIPs:
         - 10.0.0.10
       ingressVIPs:
         - 10.0.0.11
       hosts:
       - name: master-0
         role: master
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-1
         role: master
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-2
         role: master
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:03
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-0
         role: worker
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-1
         role: worker
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_public_key>'

   where:

   • replicas specifies the number of worker node replicas to match the number of baremetal worker hosts defined in the hosts section.
   • platform: baremetal specifies full hardware lifecycle management. Set to none for infrastructure-agnostic installation.
   • provisioningNetwork: Disabled specifies that the provisioning network is disabled for agent-based installations on virtual media. Nodes boot from the agent ISO, not PXE.
   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmc.address for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • disableCertificateVerification specifies whether to skip TLS certificate validation. For production deployments, configure properly signed TLS certificates and set to false. Set to true only for lab or development environments.
   • networkConfig specifies the host network configuration for each node. This example uses DHCP.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
   • bmc.address for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.

2. Create an agent-config.yaml YAML file similar to the following example:

   apiVersion: v1alpha1
   kind: AgentConfig
   metadata:
     name: my-vcp-cluster
   rendezvousIP: 10.0.0.20

   where:

   • The rendezvousIP field specifies the IP address of the first control plane node. This node coordinates the installation.

3. Generate the agent ISO by running the following command:

   $ openshift-install agent create image --dir <installation_directory>

4. Host the generated agent.x86_64.iso on an HTTP server accessible from your hosting cluster.

5. Boot each node from the agent ISO:

   1. For virtualized control plane nodes, use KubeVirt Redfish to mount the ISO and power on the VMs.
   2. For baremetal worker nodes, use the server’s BMC interface to mount the ISO and boot.

6. Monitor the installation progress by running the following command:

   $ openshift-install agent wait-for install-complete --dir <installation_directory>

   • Replace <installation_directory> with the path to the directory where the agent ISO was generated.

Verification

• After installation completes, verify the cluster is operational:

  $ export KUBECONFIG=<installation_directory>/auth/kubeconfig
  $ oc get nodes

Deploying a virtualized control plane with GitOps ZTP

Use GitOps Zero Touch Provisioning (ZTP) to deploy virtualized control plane clusters at scale. GitOps Zero Touch Provisioning (ZTP) uses GitOps to manage cluster deployments declaratively through Red Hat Advanced Cluster Management (RHACM).

Prerequisites

• RHACM is installed on a hub cluster.
• KubeVirt Redfish is installed and configured on the hosting cluster.
• Control plane VMs are created and powered off.
• A Git repository is configured for GitOps ZTP manifests.
• A pull secret is available for the cluster.

Procedure

1. Create Secret custom resources (CRs) that contain the BMC credentials for each node. The following example shows a secret for a control plane node:

   apiVersion: v1
   kind: Secret
   metadata:
     name: master-0-bmc-secret
     namespace: my-vcp-cluster
   type: Opaque
   data:
     username: <base64_encoded_username>
     password: <base64_encoded_password>

   Create similar Secret CRs for each node, for example master-1-bmc-secret, worker-0-bmc-secret and so on. For virtualized control plane nodes, use the KubeVirt Redfish credentials. For baremetal worker nodes, use the physical server’s BMC credentials.

2. Create a ClusterInstance custom resource (CR) that defines the cluster. Specify BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: my-vcp-cluster
     namespace: my-vcp-cluster
   spec:
     baseDomain: example.com
     clusterImageSetNameRef: "openshift-4.22"
     clusterName: my-vcp-cluster
     clusterType: HighlyAvailable
     platformType: BareMetal
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - cidr: 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
     apiVIPs:
       - 10.0.0.10
     ingressVIPs:
       - 10.0.0.11
     pullSecretRef:
       name: assisted-deployment-pull-secret
     sshPublicKey: "<ssh_public_key>"
     extraLabels:
       ManagedCluster:
         common: "true"
         sites: "my-vcp-cluster"
     templateRefs:
       - name: ai-cluster-templates-v1
         namespace: open-cluster-management
     nodes:
       - hostName: master-0.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
         bmcCredentialsName:
           name: master-0-bmc-secret
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: master-1.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
         bmcCredentialsName:
           name: master-1-bmc-secret
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: master-2.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
         bmcCredentialsName:
           name: master-2-bmc-secret
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: worker-0.my-vcp-cluster.example.com
         role: worker
         bmcAddress: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
         bmcCredentialsName:
           name: worker-0-bmc-secret
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: worker-1.my-vcp-cluster.example.com
         role: worker
         bmcAddress: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
         bmcCredentialsName:
           name: worker-1-bmc-secret
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management

   where:

   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • bmcAddress for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • bmcCredentialsName specifies a reference to a Secret containing BMC credentials.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmcAddress for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.

3. Commit the manifests to your Git repository.

4. Apply the configuration through ArgoCD or your GitOps tooling. ZTP generates the required resources and uses KubeVirt Redfish to provision the VMs automatically.

Verification

• Monitor the cluster deployment from the hub cluster:

  $ oc get managedcluster my-vcp-cluster
  $ oc get agentclusterinstall my-vcp-cluster -n my-vcp-cluster -o jsonpath='{.status.debugInfo.stateInfo}'

Deploying a virtualized control plane with installer-provisioned infrastructure

Use installer-provisioned infrastructure to deploy a virtualized control plane cluster. Installer-provisioned infrastructure provides full lifecycle management where the installation program automates hardware provisioning, power states, and cluster initialization.

note

Installer-provisioned infrastructure for virtualized control planes requires a provisioning network or a RHEL 9 provisioner node. Verify Installer-provisioned infrastructure support for virtualized control plane deployments with your Red Hat representative, as this combination may have additional requirements.

Prerequisites

• KubeVirt Redfish is installed and configured on the hosting cluster.
• Control plane VMs are created on the hosting cluster and powered off. In this state, the installation manages the VM power lifecycle and boot configuration through KubeVirt Redfish.
• A RHEL 9 provisioner node is available, or a provisioning network is configured.
• You have network access to the KubeVirt Redfish route.

Procedure

1. Create the install-config.yaml file with BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: v1
   metadata:
     name: my-vcp-cluster
   baseDomain: example.com
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 3
     platform:
       baremetal: {}
   compute:
     - name: worker
       architecture: amd64
       hyperthreading: Enabled
       replicas: 2
       platform:
         baremetal: {}
   networking:
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
   platform:
     baremetal:
       apiVIPs:
         - 10.0.0.10
       ingressVIPs:
         - 10.0.0.11
       provisioningNetwork: Disabled
       hosts:
       - name: master-0
         role: master
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-1
         role: master
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-2
         role: master
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:03
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-0
         role: worker
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-1
         role: worker
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_public_key>'

   where:

   • replicas specifies the number of worker node replicas to match the number of baremetal worker hosts defined in the hosts section.
   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • provisioningNetwork: Disabled specifies that the provisioning network is disabled when using virtual media.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmc.address for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • disableCertificateVerification specifies whether to skip TLS certificate validation. For production deployments, configure properly signed TLS certificates and set to false. Set to true only for lab or development environments.
   • networkConfig specifies the host network configuration for each node. This example uses DHCP.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
   • bmc.address for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.

2. Run the installation by running the following command:

   $ openshift-install create cluster --dir <installation_directory>

   The installation process uses KubeVirt Redfish to manage VM power states and boot configuration automatically.

Verification

• After installation completes, verify the cluster is operational by running the following commands:

  $ export KUBECONFIG=<installation_directory>/auth/kubeconfig
  $ oc get nodes

Additional resources

• Preparing to install with the agent-based installer
• Installation configuration parameters for the agent-based installer
• Installation configuration parameters for bare metal
• Deploying far edge sites with ZTP
• BMC addressing for installer-provisioned infrastructure
• Configuring KubeVirt Redfish for VM management
• Preparing the provisioner node for OpenShift Container Platform installation