Deploying hosted control planes on bare metal with the Agent platform¶

To maximize hardware performance and maintain control over your physical infrastructure, you can deploy hosted control planes on bare metal by using the Agent platform. This deployment method reduces virtualization overhead and offers low-latency networking for performance-intensive workloads.

About hosted control planes on bare metal with the Agent platform¶

You can deploy hosted control planes on bare-metal infrastructure with the Agent platform by configuring an OpenShift Container Platform cluster to function as a management cluster.

The management cluster is the OpenShift Container Platform cluster where the control planes are hosted. In some contexts, the management cluster is also known as the hosting cluster. The management cluster is not the same thing as the managed cluster. A managed cluster is a cluster that the hub cluster manages.

The hosted control planes feature is enabled by default.

The multicluster engine Operator supports only the default local-cluster, which is a hub cluster that is managed, and the hub cluster as the management cluster. If you have Red Hat Advanced Cluster Management installed, you can use the managed hub cluster, also known as the local-cluster, as the management cluster.

A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the management cluster. The hosted cluster includes the control plane and its corresponding data plane. You can use the multicluster engine Operator console or the hosted control plane command-line interface (hcp) to create a hosted cluster.

The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine Operator".

Additional resources

Disabling the automatic import of hosted clusters into multicluster engine Operator

Preparing to deploy hosted control planes on bare metal¶

Before you deploy hosted control planes on bare metal, ensure that you understand the requirements for the deployment.

Run the management cluster and compute nodes on the same platform.
All bare-metal hosts require a manual start with a Discovery Image ISO that the central infrastructure management provides. You can start the hosts manually or through automation by using the Cluster Baremetal Operator. After each host starts, it runs an Agent process to discover the host details and complete the installation. An Agent custom resource represents each host.
When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using logical volume manager storage".

Additional resources

Bare metal firewall, port, and service requirements¶

You must meet the firewall, port, and service requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters.

Note

Services run on their default ports. However, if you use the NodePort publishing strategy, services run on the port that is assigned by the NodePort service.

Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. For production deployments, use a load balancer to simplify access through a single IP address.

If your hub cluster has a proxy configuration, ensure that it can reach the hosted cluster API endpoint by adding all hosted cluster API endpoints to the noProxy field on the Proxy object. For more information, see "Configuring the cluster-wide proxy".

A hosted control plane exposes the following services on bare metal:

APIServer
- The APIServer service runs on port 6443 by default and requires ingress access for communication between the control plane components.
- If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses.
OAuthServer
- The OAuthServer service runs on port 443 by default when you use the route and ingress to expose the service.
- If you use the NodePort publishing strategy, use a firewall rule for the OAuthServer service.
Konnectivity
- The Konnectivity service runs on port 443 by default when you use the route and ingress to expose the service.
- The Konnectivity agent establishes a reverse tunnel to allow the control plane to access the network for the hosted cluster. The agent uses egress to connect to the Konnectivity server. The server is exposed by using either a route on port 443 or a manually assigned NodePort.
- If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443.
- If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes.
Ignition
- The Ignition service runs on port 443 by default when you use the route and ingress to expose the service.
- If you use the NodePort publishing strategy, use a firewall rule for the Ignition service.

You do not need the following services on bare metal:

OVNSbDb
OIDC

Additional resources

Configuring the cluster-wide proxy

Bare metal infrastructure requirements¶

Although the Agent platform does not create any infrastructure, the Agent platform does have requirements for infrastructure.

Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node.
DNS: The API and ingress endpoints must be routable.

Configuring a management cluster for bare metal¶

Before you create a hosted cluster on bare metal with the Agent platform, you need a properly configured OpenShift Container Platform management cluster.

Prerequisites

The management cluster and compute nodes must be on the same platform.
You need the multicluster engine for Kubernetes Operator 2.2 and later installed on an OpenShift Container Platform cluster. You can install multicluster engine Operator as an Operator from the OpenShift Container Platform software catalog.

Procedure

The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported in multicluster engine Operator 2.2 and later. For more information about the local-cluster, see "Advanced configuration" in the Red Hat Advanced Cluster Management documentation. You can check the status of your hub cluster by running the following command:
```
$ oc get managedclusters local-cluster
```
Add the topology.kubernetes.io/zone label to your bare-metal hosts on your management cluster. Ensure that each host has a unique value for topology.kubernetes.io/zone. Otherwise, all of the control plane pods are scheduled on a single node, causing a single point of failure.
Use the Agent platform to provision hosted control planes on bare metal. The Agent platform uses the central infrastructure management service to add compute nodes to a hosted cluster. For more information, see "Enabling the central infrastructure management service".
Install the hosted control plane command-line interface. For more information, see "Installing the hosted control planes command-line interface".

Additional resources

DNS configurations on bare metal¶

The API Server for the hosted cluster is exposed as a NodePort service. In production environments, use a load balancer in front of the API. For example, you can use an external load balancer or a Kubernetes service type load balancer in combination with MetalLB.

The DNS entries must point to the load balancer to forward incoming traffic to the API and ingress.

Example DNS configuration for a connected environment on an IPv4 network

api.cluster.company.example     IN A <ipv4_of_api_load_balancer>
api-int.cluster.company.example IN A <ipv4_of_api_load_balancer>
*.apps.cluster.company.example  IN A <ipv4_of_ingress_load_balancer>

Example DNS configuration for a disconnected environment on an IPv6 network

api.cluster.company.example     IN AAAA <ipv6_of_api_load_balancer>
api-int.cluster.company.example IN AAAA <ipv6_of_api_load_balancer>
*.apps.cluster.company.example  IN AAAA <ipv6_of_ingress_load_balancer>

If you have a dual-stack setup, you must include DNS entries for both IPv4 and IPv6.

Defining a custom DNS name¶

As a cluster administrator, you can create a hosted cluster with an external API DNS name that differs from the internal endpoint that gets used for node bootstraps and control plane communication.

You might want to define a different DNS name for the following reasons:

To replace the user-facing TLS certificate with one from a public CA without breaking the control plane functions that bind to the internal root CA
To support split-horizon DNS and NAT scenarios
To ensure a similar experience to standalone control planes, where you can use functions, such as the Show Login Command function, with the correct kubeconfig and DNS configuration

You can define a DNS name either during your initial setup or during postinstallation operations, by entering a domain name in the kubeAPIServerDNSName parameter of a HostedCluster object.

Prerequisites

You have a valid TLS certificate that covers the DNS name that you set in the kubeAPIServerDNSName parameter.
You have a resolvable DNS name URI that can reach and point to the correct address.

Procedure

In the specification for the HostedCluster object, add the kubeAPIServerDNSName parameter and the address for the domain and specify which certificate to use, as shown in the following example:
```
#...
spec:
  configuration:
    apiServer:
      servingCerts:
        namedCertificates:
        - names:
          - xxx.example.com
          - yyy.example.com
          servingCertificate:
            name: <my_serving_certificate>
  kubeAPIServerDNSName: <custom_address>
```
The value for the kubeAPIServerDNSName parameter must be a valid and addressable domain.

After you define the kubeAPIServerDNSName parameter and specify the certificate, the Control Plane Operator controllers create a kubeconfig file named custom-admin-kubeconfig, where the file gets stored in the HostedControlPlane namespace. The generation of certificates happen from the root CA, and the HostedControlPlane namespace manages their expiration and renewal.

The Control Plane Operator reports a new kubeconfig file named CustomKubeconfig in the HostedControlPlane namespace. That file uses the defined new server in the kubeAPIServerDNSName parameter.

A reference for the custom kubeconfig file exists in the status parameter as CustomKubeconfig of the HostedCluster object. The CustomKubeConfig parameter is optional, and you can add the parameter only if the kubeAPIServerDNSName parameter is not empty. After you set the CustomKubeConfig parameter, the parameter triggers the generation of a secret named <hosted_cluster_name>-custom-admin-kubeconfig in the HostedCluster namespace. You can use the secret to access the HostedCluster API server. If you remove the CustomKubeConfig parameter during postinstallation operations, deletion of all related secrets and status references occur.

Note

Defining a custom DNS name does not directly impact the data plane, so no expected rollouts occur. The HostedControlPlane namespace receives the changes from the HyperShift Operator and deletes the corresponding parameters.

If you remove the kubeAPIServerDNSName parameter from the specification for the HostedCluster object, all newly generated secrets and the CustomKubeconfig reference are removed from the cluster and from the status parameter.

Creating an InfraEnv resource¶

Before you can create a hosted cluster on bare metal, you need an InfraEnv resource.

On hosted control planes, the control-plane components run as pods on the management cluster while the data plane runs on dedicated nodes. You can use the Assisted Service to boot your hardware with a discovery ISO that adds your hardware to a hardware inventory.

Later, when you create a hosted cluster, the hardware from the inventory is used to provision the data-plane nodes. The object that is used to get the discovery ISO is an InfraEnv resource. You need to create a BareMetalHost object that configures the cluster to boot the bare-metal node from the discovery ISO.

Creating an InfraEnv resource and adding nodes¶

To ensure that your hardware is provisioned correctly before you create a hosted cluster on bare metal, create an InfraEnv resource. You can create the resource and add nodes by using the command-line interface (CLI).

Procedure

Create a namespace to store your hardware inventory by entering the following command:
```
$ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig create \
  namespace <namespace_example>
```
where:

<directory_example>

Is the name of the directory where the kubeconfig file for the management cluster is saved.

<namespace_example>

Is the name of the namespace that you are creating; for example, hardware-inventory.
Example output
```
namespace/hardware-inventory created
```

Copy the pull secret of the management cluster by entering the following command:

$ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig \
  -n openshift-config get secret pull-secret -o yaml \
  | grep -vE "uid|resourceVersion|creationTimestamp|namespace" \
  | sed "s/openshift-config/<namespace_example>/g" \
  | oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig \
  -n <namespace> apply -f -

where:

<directory_example>: Is the name of the directory where the kubeconfig file for the management cluster is saved.
<namespace_example>: Is the name of the namespace that you are creating; for example, hardware-inventory.

Example output

secret/pull-secret created

Create the InfraEnv resource by adding the following content to a YAML file:

apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: hosted
  namespace: <namespace_example>
spec:
  additionalNTPSources:
  - <ip_address>
  pullSecretRef:
    name: pull-secret
  sshAuthorizedKey: <ssh_public_key>
# ...

Apply the changes to the YAML file by entering the following command:
```
$ oc apply -f <infraenv_config>.yaml
```
Replace <infraenv_config> with the name of your file.

Verify that the InfraEnv resource was created by entering the following command:

$ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig \
  -n <namespace_example> get infraenv hosted

Add bare-metal hosts by following one of two methods:

If you do not use the Metal3 Operator, obtain the discovery ISO from the InfraEnv resource and boot the hosts manually by completing the following steps:

Download the live ISO by entering the following commands:

$ oc get infraenv -A

$ oc get infraenv <namespace_example> -o jsonpath='{.status.isoDownloadURL}' -n <namespace_example> <iso_url>

Boot the ISO. The node communicates with the Assisted Service and registers as an agent in the same namespace as the InfraEnv resource.

For each agent, set the installation disk ID and hostname, and approve it to indicate that the agent is ready for use.

Enter the following command to get the agents for your hosted control plane namespace:

$ oc -n <hosted_control_plane_namespace> get agents

In this example, two agents are listed.

Example output

NAME                                   CLUSTER   APPROVED   ROLE          STAGE
example-agent-1                        auto-assign
example-agent-2                        auto-assign

Enter the following command to set the installation disk ID and hostname for the first agent:

$ oc -n <hosted_control_plane_namespace> \
  patch agent example-agent-1 \
  -p '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-0.example.krnl.es"}}' \
  --type merge

Enter the following command to set the installation disk ID and hostname for the second agent:

$ oc -n <hosted_control_plane_namespace> \
  patch agent example-agent-2 -p \
  '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-1.example.krnl.es"}}' \
  --type merge

If you use the Metal3 Operator, you can automate the bare-metal host registration by creating the following objects:

Create a YAML file and add the following content to it:

apiVersion: v1
kind: Secret
metadata:
  name: hosted-worker0-bmc-secret
  namespace: <namespace_example>
data:
  password: <password>
  username: <username>
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: hosted-worker1-bmc-secret
  namespace: <namespace_example>
data:
  password: <password>
  username: <username>
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: hosted-worker2-bmc-secret
  namespace: <namespace_example>
data:
  password: <password>
  username: <username>
type: Opaque
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: hosted-worker0
  namespace: <namespace_example>
  labels:
    infraenvs.agent-install.openshift.io: hosted
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: hosted-worker0
spec:
  automatedCleaningMode: disabled
  bmc:
    disableCertificateVerification: True
    address: <bmc_address>
    credentialsName: hosted-worker0-bmc-secret
  bootMACAddress: aa:aa:aa:aa:02:01
  online: true
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: hosted-worker1
  namespace: <namespace_example>
  labels:
    infraenvs.agent-install.openshift.io: hosted
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: hosted-worker1
spec:
  automatedCleaningMode: disabled
  bmc:
    disableCertificateVerification: True
    address: <bmc_address>
    credentialsName: hosted-worker1-bmc-secret
  bootMACAddress: aa:aa:aa:aa:02:02
  online: true
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: hosted-worker2
  namespace: <namespace_example>
  labels:
    infraenvs.agent-install.openshift.io: hosted
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: hosted-worker2
spec:
  automatedCleaningMode: disabled
  bmc:
    disableCertificateVerification: True
    address: <bmc_address>
    credentialsName: hosted-worker2-bmc-secret
  bootMACAddress: aa:aa:aa:aa:02:03
  online: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: capi-provider-role
  namespace: <namespace_example>
rules:
- apiGroups:
  - agent-install.openshift.io
  resources:
  - agents
  verbs:
  - '*'

where:

<namespace_example>: Is the your namespace.
<password>: Is the password for your secret.
<username>: Is the user name for your secret.
<bmc_address>: Is the BMC address for the BareMetalHost object.

Note

When you apply this YAML file, the following objects are created:

Secrets with credentials for the Baseboard Management Controller (BMCs)
The BareMetalHost objects
A role for the HyperShift Operator to be able to manage the agents

Notice how the InfraEnv resource is referenced in the BareMetalHost objects by using the infraenvs.agent-install.openshift.io: hosted custom label. This ensures that the nodes are booted with the ISO generated.

Apply the changes to the YAML file by entering the following command:
```
$ oc apply -f <bare_metal_host_config>.yaml
```
Replace <bare_metal_host_config> with the name of your file.

Enter the following command, and then wait a few minutes for the BareMetalHost objects to move to the Provisioning state:

$ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig -n <namespace_example> get bmh

Example output

NAME             STATE          CONSUMER   ONLINE   ERROR   AGE
hosted-worker0   provisioning              true             106s
hosted-worker1   provisioning              true             106s
hosted-worker2   provisioning              true             106s

Enter the following command to verify that nodes are booting and showing up as agents. This process can take a few minutes, and you might need to enter the command more than once.

$ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig -n <namespace_example> get agent

Example output

NAME                                   CLUSTER   APPROVED   ROLE          STAGE
aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0201             true       auto-assign
aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0202             true       auto-assign
aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0203             true       auto-assign

Creating an InfraEnv resource by using the console¶

If you prefer to use the OpenShift Container Platform web console, you can use it to create an InfraEnv resource for a hosted cluster on bare metal.

Procedure

Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see "Accessing the web console".
In the console header, ensure that All Clusters is selected.
Click Infrastructure → Host inventory → Create infrastructure environment.
After you create the InfraEnv resource, add bare-metal hosts from within the InfraEnv view by clicking Add hosts and selecting from the available options.

Additional resources

Accessing the web console

Creating a hosted cluster on bare metal¶

You can create a hosted cluster on bare metal with the Agent platform by using the command-line interface (CLI), the console, or by using a mirror registry.

Creating a hosted cluster by using the CLI¶

On bare-metal infrastructure, you can import a hosted cluster or create one by using the command-line interface (CLI).

After you enable the Assisted Installer as an add-on to multicluster engine Operator and you create a hosted cluster with the Agent platform, the HyperShift Operator installs the Agent Cluster API provider in the hosted control plane namespace. The Agent Cluster API provider connects a management cluster that hosts the control plane and a hosted cluster that consists of only the compute nodes.

Prerequisites

Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster. Otherwise, the multicluster engine Operator cannot manage the hosted cluster.
Do not use the word clusters as a hosted cluster name.
You cannot create a hosted cluster in the namespace of a multicluster engine Operator managed cluster.
For best security and management practices, create a hosted cluster separate from other hosted clusters.
Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending persistent volume claims (PVCs).
By default when you use the hcp create cluster agent command, the command creates a hosted cluster with configured node ports. The preferred publishing strategy for hosted clusters on bare metal exposes services through a load balancer. If you create a hosted cluster by using the web console or by using Red Hat Advanced Cluster Management, to set a publishing strategy for a service besides the Kubernetes API server, you must manually specify the servicePublishingStrategy information in the HostedCluster custom resource.
Ensure that you meet the requirements described in "Requirements for hosted control planes on bare metal", which includes requirements related to infrastructure, firewalls, ports, and services. For example, those requirements describe how to add the appropriate zone labels to the bare-metal hosts in your management cluster, as shown in the following example commands:
```
$ oc label node [compute-node-1] topology.kubernetes.io/zone=zone1
```
```
$ oc label node [compute-node-2] topology.kubernetes.io/zone=zone2
```
```
$ oc label node [compute-node-3] topology.kubernetes.io/zone=zone3
```
Ensure that you have added bare-metal nodes to a hardware inventory.

Procedure

Create a namespace by entering the following command:
```
$ oc create ns <hosted_cluster_namespace>
```
Replace <hosted_cluster_namespace> with an identifier for your hosted cluster namespace. The HyperShift Operator creates the namespace. During the hosted cluster creation process on bare-metal infrastructure, a generated Cluster API provider role requires that the namespace already exists.
Create the configuration file for your hosted cluster by entering the following command:
```
$ hcp create cluster agent \
  --name=<hosted_cluster_name> \
  --pull-secret=<path_to_pull_secret> \
  --agent-namespace=<hosted_control_plane_namespace> \
  --base-domain=<base_domain> \
  --api-server-address=api.<hosted_cluster_name>.<base_domain> \
  --etcd-storage-class=<etcd_storage_class> \
  --ssh-key=<path_to_ssh_key> \
  --namespace=<hosted_cluster_namespace> \
  --control-plane-availability-policy=HighlyAvailable \
  --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image>-multi \
  --node-pool-replicas=<node_pool_replica_count> \
  --disable-cluster-capabilities=<capability> \
  --enable-cluster-capabilities=<capability> \
  --render \
  --render-sensitive > hosted-cluster-config.yaml
```
where:
- --name specifies the name of your hosted cluster, such as example.
- --pull-secret specifies the path to your pull secret, such as /user/name/pullsecret.
- --agent-namespace specifies your hosted control plane namespace, such as clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.
- --base-domain specifies your base domain, such as krnl.es.
- --api-server-address specifies the IP address that gets used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
- --etcd-storage-class specifies the etcd storage class name, such as lvm-storageclass.
- --ssh-key specifies the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
- --namespace specifies your hosted cluster namespace.
- --control-plane-availability-policy specifies the availability policy for the hosted control plane components. Supported options are SingleReplica and HighlyAvailable. The default value is HighlyAvailable.
- --release-image specifies the supported OpenShift Container Platform version that you want to use, such as 4.22.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the release image digest".
- --node-pool-replicas specifies the node pool replica count, such as 3. You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, you do not create node pools.
- --disable-cluster-capabilities specifies that you want to disable optional capabilities in the hosted cluster. This flag is optional. For more information, see "Capabilities for hosted clusters".
- --enable-cluster-capabilities specifies that you want to enable optional capabilities in the hosted cluster. This flag is optional. For more information, see "Capabilities for hosted clusters".
- --render renders the output as YAML to stdout instead of applying the resources to the cluster. By default, secrets are not included in the rendered output.
- --render-sensitive includes secrets in the rendered output when used with the --render flag.
Configure the service publishing strategy. By default, hosted clusters use the NodePort service publishing strategy because node ports are always available without additional infrastructure. However, you can configure the service publishing strategy to use a load balancer.
- If you are using the default NodePort strategy, configure the DNS to point to the hosted cluster compute nodes, not the management cluster nodes. For more information, see "DNS configurations on bare metal".
- For production environments, use the LoadBalancer strategy because this strategy provides certificate handling and automatic DNS resolution. The following example demonstrates changing the service publishing LoadBalancer strategy in your hosted cluster configuration file:
```
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
# ...
spec:
  services:
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: OIDC
    servicePublishingStrategy:
      type: Route
  sshKey:
    name: <ssh_key>
# ...
```
  Specify LoadBalancer as the API Server type. For all other services, specify Route as the type.
If you use external load balancers, configure the ingress endpoint as shown in the following example. If you do not configure the endpoint, the default behavior is to randomize the node port that the service exposes the ingress on. To configure how the ingress controller publishes the default ingress route, set the endpointPublishingStrategy parameter and its underlying functions by editing the HostedCluster resource:
```
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
#...
spec:
  operatorConfiguration:
    ingressOperator:
      endpointPublishingStrategy:
        hostNetwork:
          httpPort: 80
          httpsPort: 443
          protocol: TCP
          statsPort: 1936
        type: HostNetwork
#...
```
The spec.operatorConfiguration.ingressOperator.endPointPublishingStrategy.type parameter specifies the endpoint for the load balancer. For bare-metal installations, use the HostNetwork type.
Apply the changes to the hosted cluster configuration file by entering the following command:
```
$ oc apply -f hosted_cluster_config.yaml
```

Check for the creation of the hosted cluster, node pools, and pods by entering the following commands:

$ oc get hostedcluster \
  <hosted_cluster_namespace> -n \
  <hosted_cluster_namespace> -o \
  jsonpath='{.status.conditions[?(@.status=="False")]}' | jq .

$ oc get nodepool \
  <hosted_cluster_namespace> -n \
  <hosted_cluster_namespace> -o \
  jsonpath='{.status.conditions[?(@.status=="False")]}' | jq .

$ oc get pods -n <hosted_cluster_namespace>

Confirm that the hosted cluster is ready. The status of Available: True indicates the readiness of the cluster and the node pool status shows AllMachinesReady: True. These statuses indicate the healthiness of all cluster Operators.

Install MetalLB in the hosted cluster:

Extract the kubeconfig file from the hosted cluster and set the environment variable for hosted cluster access by entering the following commands:

$ oc get secret \
  <hosted_cluster_namespace>-admin-kubeconfig \
  -n <hosted_cluster_namespace> \
  -o jsonpath='{.data.kubeconfig}' \
  | base64 -d > \
  kubeconfig-<hosted_cluster_namespace>.yaml

$ export KUBECONFIG="/path/to/kubeconfig-<hosted_cluster_namespace>.yaml"

Install the MetalLB Operator by creating the install-metallb-operator.yaml file:

apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: metallb-operator
  namespace: metallb-system
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: metallb-operator
  namespace: metallb-system
spec:
  channel: "stable"
  name: metallb-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
# ...

Apply the file by entering the following command:
```
$ oc apply -f install-metallb-operator.yaml
```

Configure the MetalLB IP address pool by creating the deploy-metallb-ipaddresspool.yaml file:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: metallb
  namespace: metallb-system
spec:
  autoAssign: true
  addresses:
  - 10.11.176.71-10.11.176.75
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - metallb
# ...

Apply the configuration by entering the following command:
```
$ oc apply -f deploy-metallb-ipaddresspool.yaml
```
Verify the installation of MetalLB by checking the Operator status, the IP address pool, and the L2Advertisement resource by entering the following commands:
```
$ oc get pods -n metallb-system
```
```
$ oc get ipaddresspool -n metallb-system
```
```
$ oc get l2advertisement -n metallb-system
```

Configure the load balancer for ingress:

Create the ingress-loadbalancer.yaml file:

apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/address-pool: metallb
  name: metallb-ingress
  namespace: openshift-ingress
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
  type: LoadBalancer
# ...

Apply the configuration by entering the following command:
```
$ oc apply -f ingress-loadbalancer.yaml
```

Verify that the load balancer service works as expected by entering the following command:

$ oc get svc metallb-ingress -n openshift-ingress

Example output

NAME              TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
metallb-ingress   LoadBalancer   172.31.127.129   10.11.176.71   80:30961/TCP,443:32090/TCP   16h

Configure the DNS to work with the load balancer:

Configure the DNS for the apps domain by pointing the *.apps.<hosted_cluster_namespace>.<base_domain> wildcard DNS record to the load balancer IP address.

Verify the DNS resolution by entering the following command:

$ nslookup console-openshift-console.apps.<hosted_cluster_namespace>.<base_domain> <load_balancer_ip_address>

Example output

Server:         10.11.176.1
Address:        10.11.176.1#53

Name:   console-openshift-console.apps.my-hosted-cluster.sample-base-domain.com
Address: 10.11.176.71

Verification

Check the cluster Operators by entering the following command:
```
$ oc get clusteroperators
```
Ensure that all Operators show AVAILABLE: True, PROGRESSING: False, and DEGRADED: False.
Check the nodes by entering the following command:
```
$ oc get nodes
```
Ensure that each node has the READY status.

Test access to the console by entering the following URL in a web browser:

https://console-openshift-console.apps.<hosted_cluster_namespace>.<base_domain>

Additional resources

Creating a hosted cluster on bare metal by using the console¶

Instead of using the command-line interface to create a hosted cluster on bare metal, you can also use the OpenShift Container Platform web console.

Procedure

Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see "Accessing the web console".
In the console header, ensure that All Clusters is selected.
Click Infrastructure → Clusters.
Click Create cluster → Host inventory → Hosted control plane.

The Create cluster page is displayed.
On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation.
Note

As you enter details about the cluster, you might find the following tips useful:
- If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see "Creating a credential for an on-premises environment".
- On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated.
- On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace.
- On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods.
Review your entries and click Create.

The Hosted cluster view is displayed.
Monitor the deployment of the hosted cluster in the Hosted cluster view.
If you do not see information about the hosted cluster, ensure that All Clusters is selected, then click the cluster name.
Wait until the control plane components are ready. This process can take a few minutes.
To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Additional resources

Creating a hosted cluster on bare metal by using a mirror registry¶

You can use a mirror registry to create a hosted cluster on bare metal by specifying the --image-content-sources flag in the hcp create cluster command.

Procedure

Create a YAML file to define Image Content Source Policies (ICSP). See the following example:

- mirrors:
  - brew.registry.redhat.io
  source: registry.redhat.io
- mirrors:
  - brew.registry.redhat.io
  source: registry.stage.redhat.io
- mirrors:
  - brew.registry.redhat.io
  source: registry-proxy.engineering.redhat.com

Save the file as icsp.yaml. This file contains your mirror registries.
To create a hosted cluster by using your mirror registries, run the following command:
```
$ hcp create cluster agent \
    --name=<hosted_cluster_name> \
    --pull-secret=<path_to_pull_secret> \
    --agent-namespace=<hosted_control_plane_namespace> \
    --base-domain=<base_domain> \
    --api-server-address=api.<hosted_cluster_name>.<base_domain> \
    --image-content-sources <icsp.yaml>  \
    --ssh-key  <path_to_ssh_key> \
    --namespace <hosted_cluster_namespace> \
    --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image>
```
where:

<hosted_cluster_name>

Specifies the name of your hosted cluster, for example, my-hosted-cluster.

<path_to_pull_secret>

Specifies the path to your pull secret, for example, /user/name/pullsecret.

<hosted_control_plane_namespace>

Specifies your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

<base_domain>

Specifies your base domain, for example, krnl.es.

<hosted_cluster_name>.<base_domain>

Specifies the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

<icsp.yaml>

Specifies the icsp.yaml file that defines ICSP and your mirror registries.

<path_to_ssh_key>

Specifies the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

<hosted_cluster_namespace>

Specifies your hosted cluster namespace.

<ocp_release_image>

Specifies the supported OpenShift Container Platform version that you want to use, for example, 4.22.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the release image digest".

Additional resources

Capabilities for hosted clusters¶

To reduce resource consumption and prevent unnecessary Operators and operands from being deployed, administrators can enable or disable optional OpenShift Container Platform components when they create a hosted cluster.

When capabilities are not specified on a HostedCluster resource, the cluster uses the OpenShift Container Platform version’s DefaultCapabilitySet settings, excluding the baremetal capability. As a result, most optional components are enabled by default.

Warning

Capabilities are immutable after cluster creation. You cannot change them after you create the HostedCluster resource.

Capabilities that you can enable or disable for a hosted cluster¶

Familiarize yourself with the supported capabilities that you can enable or disable for a HostedCluster resource.

The capabilities are described in the following table:

Capability	Description
`ImageRegistry`	The OpenShift Image Registry Operator and its operands, including cloud storage infrastructure, such as S3 buckets and Identity and Access Management (IAM) users.
`openshift-samples`	The OpenShift Samples Operator, which manages example `ImageStreams` and templates.
`Insights`	The Insights Operator, which collects and uploads cluster telemetry data.
`baremetal`	The Bare Metal Infrastructure Operator. This capability is excluded from the default set. If needed, you must explicitly enable it.
`Console`	The OpenShift Web Console Operator and its operands.
`NodeTuning`	The Node Tuning Operator, which manages node-level performance tuning by using TuneD and performance profiles.
`Ingress`	The OpenShift Ingress Operator, which manages the default router of the cluster.

The following rules apply when you combine capability settings:

No overlap: A capability cannot be in both the enabled and disabled lists simultaneously.
Console requires Ingress: You can disable the Ingress capability only if the Console capability is also disabled because the console depends on Ingress.
Version requirement: You must use OpenShift Container Platform 4.20 or later to disable any of the following capabilities: openshift-samples, Insights, Console, NodeTuning, and Ingress. You can disable ImageRegistry and baremetal on versions earlier than 4.20.
Bare metal default exclusion: The baremetal capability is excluded from the default set. You can add it to the cluster by explicitly enabling it.

Setting capabilities for a hosted cluster¶

To reduce unnecessary resource consumption, you can control which optional capabilities are enabled for a hosted cluster when you create the cluster.

Warning

Capabilities are immutable after cluster creation. You cannot change them after you create the HostedCluster resource.

You can specify which capabilities are enabled by either using the hcp command-line interface (CLI) or by setting the HostedCluster manifest.

Procedure

To specify which capabilities are enabled in a hosted cluster by using the CLI, you can add the --disable-cluster-capabilities flag, the --enable-cluster-capabilities flag, or both. The following example shows how to disable the ImageRegistry, Console, and Ingress capabilities and enable the baremetal capability while you create a hosted cluster on AWS by using the hcp command-line interface:
```
$ hcp create cluster aws \
    --name my-hosted-cluster \
    --disable-cluster-capabilities=ImageRegistry,Console,Ingress \
    --enable-cluster-capabilities=baremetal
```
You can specify multiple capabilities as a comma-separated list. The supported values are as follows:
- ImageRegistry
- openshift-samples
- Insights
- baremetal
- Console
- NodeTuning
- Ingress

To specify which capabilities are enabled in a hosted cluster by using the HostedCluster manifest at cluster creation time, see the following examples:

To directly disable capabilities in a hosted cluster, add the spec.capabilities.disabled section in the HostedCluster resource:

apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: my-hosted-cluster
  namespace: my-cluster-namespace
spec:
  capabilities:
    disabled:
      - ImageRegistry
      - Console
      - Ingress
  # ...

To explicitly enable a capability that is not part of the default set of capabilities, such as the baremetal capability, see the following example:

apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: my-hosted-cluster
  namespace: my-cluster-namespace
spec:
  capabilities:
    enabled:
      - baremetal
  # ...

You can use both enabled and disabled if no capabilities are in both lists. See the following example:

apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: my-hosted-cluster
  namespace: my-cluster-namespace
spec:
  capabilities:
    enabled:
      - baremetal
    disabled:
      - ImageRegistry
      - openshift-samples
  # ...

Verifying hosted cluster creation¶

After the deployment process is complete, you can verify that the hosted cluster was created successfully.

After you create the hosted cluster, wait a few minutes before you start the steps in the procedure.

Procedure

Obtain the kubeconfig for your new hosted cluster by entering the extract command:

$ oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig \
  --to=- > kubeconfig-<hosted-cluster-name>

Use the kubeconfig to view the cluster Operators of the hosted cluster. Enter the following command:

$ oc get co --kubeconfig=kubeconfig-<hosted-cluster-name>

Example output

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    4.10.26   True        False         False      2m38s
dns                                        4.10.26   True        False         False      2m52s
image-registry                             4.10.26   True        False         False      2m8s
ingress                                    4.10.26   True        False         False      22m

You can also view the running pods on your hosted cluster by entering the following command:

$ oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name>

Example output

NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
openshift-cluster-node-tuning-operator             tuned-dhw5p                                               1/1     Running            0               109s
openshift-cluster-storage-operator                 cluster-storage-operator-5f784969f5-vwzgz                 1/1     Running            1 (113s ago)    20m
openshift-cluster-storage-operator                 csi-snapshot-controller-6b7687b7d9-7nrfw                  1/1     Running            0               3m8s
openshift-console                                  console-5cbf6c7969-6gk6z                                  1/1     Running            0               119s
openshift-console                                  downloads-7bcd756565-6wj5j                                1/1     Running            0               4m3s
openshift-dns-operator                             dns-operator-77d755cd8c-xjfbn                             2/2     Running            0               21m
openshift-dns                                      dns-default-kfqnh                                         2/2     Running            0               113s