--- title: Installing managed clusters with RHACM and ClusterInstance resources --- # Installing managed clusters with RHACM and ClusterInstance resources {#ztp-deploying-far-edge-sites} You can provision OpenShift Container Platform clusters at scale with Red Hat Advanced Cluster Management (RHACM) using the assisted service and the GitOps plugin policy generator with core-reduction technology enabled. The GitOps Zero Touch Provisioning (ZTP) pipeline performs the cluster installations. GitOps ZTP can be used in a disconnected environment. > [!IMPORTANT] > Using `PolicyGenTemplate` CRs to manage and deploy policies to managed clusters will be deprecated in an upcoming OpenShift Container Platform release. Equivalent and improved functionality is available using Red Hat Advanced Cluster Management (RHACM) and `PolicyGenerator` CRs. > > For more information about `PolicyGenerator` resources, see the RHACM [Integrating Policy Generator](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/governance/index#integrate-policy-generator) documentation. **Additional resources** {._additional-resources} - [Configuring managed cluster policies by using PolicyGenerator resources](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator#ztp-configuring-managed-clusters-policygenerator) - [Comparing RHACM PolicyGenerator and PolicyGenTemplate resource patching](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator#ztp-comparing-pgt-and-rhacm-pg-patching-strategies_ztp-configuring-managed-clusters-policygenerator) ## GitOps ZTP and Topology Aware Lifecycle Manager {#ztp-talo-integration_ztp-deploying-far-edge-sites} GitOps Zero Touch Provisioning (ZTP) generates installation and configuration CRs from manifests stored in Git. These artifacts are applied to a centralized hub cluster where Red Hat Advanced Cluster Management (RHACM), the assisted service, and the Topology Aware Lifecycle Manager (TALM) use the CRs to install and configure the managed cluster. The configuration phase of the GitOps ZTP pipeline uses the TALM to orchestrate the application of the configuration CRs to the cluster. There are several key integration points between GitOps ZTP and the TALM. Inform policies : By default, GitOps ZTP creates all policies with a remediation action of `inform`. These policies cause RHACM to report on compliance status of clusters relevant to the policies but does not apply the desired configuration. During the GitOps ZTP process, after OpenShift installation, the TALM steps through the created `inform` policies and enforces them on the target managed cluster(s). This applies the configuration to the managed cluster. Outside of the GitOps ZTP phase of the cluster lifecycle, this allows you to change policies without the risk of immediately rolling those changes out to affected managed clusters. You can control the timing and the set of remediated clusters by using TALM. Automatic creation of ClusterGroupUpgrade CRs : To automate the initial configuration of newly deployed clusters, TALM monitors the state of all `ManagedCluster` CRs on the hub cluster. Any `ManagedCluster` CR that does not have a `ztp-done` label applied, including newly created `ManagedCluster` CRs, causes the TALM to automatically create a `ClusterGroupUpgrade` CR with the following characteristics: - The `ClusterGroupUpgrade` CR is created and enabled in the `ztp-install` namespace. - `ClusterGroupUpgrade` CR has the same name as the `ManagedCluster` CR. - The cluster selector includes only the cluster associated with that `ManagedCluster` CR. - The set of managed policies includes all policies that RHACM has bound to the cluster at the time the `ClusterGroupUpgrade` is created. - Pre-caching is disabled. - Timeout set to 4 hours (240 minutes). The automatic creation of an enabled `ClusterGroupUpgrade` ensures that initial zero-touch deployment of clusters proceeds without the need for user intervention. Additionally, the automatic creation of a `ClusterGroupUpgrade` CR for any `ManagedCluster` without the `ztp-done` label allows a failed GitOps ZTP installation to be restarted by simply deleting the `ClusterGroupUpgrade` CR for the cluster. Waves : Each policy generated from a `PolicyGenerator` or `PolicyGentemplate` CR includes a `ztp-deploy-wave` annotation. This annotation is based on the same annotation from each CR which is included in that policy. The wave annotation is used to order the policies in the auto-generated `ClusterGroupUpgrade` CR. The wave annotation is not used other than for the auto-generated `ClusterGroupUpgrade` CR. > [!NOTE] > All CRs in the same policy must have the same setting for the `ztp-deploy-wave` annotation. The default value of this annotation for each CR can be overridden in the `PolicyGenerator` or `PolicyGentemplate`. The wave annotation in the source CR is used for determining and setting the policy wave annotation. This annotation is removed from each built CR which is included in the generated policy at runtime. The TALM applies the configuration policies in the order specified by the wave annotations. The TALM waits for each policy to be compliant before moving to the next policy. It is important to ensure that the wave annotation for each CR takes into account any prerequisites for those CRs to be applied to the cluster. For example, an Operator must be installed before or concurrently with the configuration for the Operator. Similarly, the `CatalogSource` for an Operator must be installed in a wave before or concurrently with the Operator Subscription. The default wave value for each CR takes these prerequisites into account. > [!NOTE] > Multiple CRs and policies can share the same wave number. Having fewer policies can result in faster deployments and lower CPU usage. It is a best practice to group many CRs into relatively few waves. To check the default wave value in each source CR, run the following command against the `out/source-crs` directory that is extracted from the `ztp-site-generate` container image: ```terminal $ grep -r "ztp-deploy-wave" out/source-crs ``` Phase labels : The `ClusterGroupUpgrade` CR is automatically created and includes directives to annotate the `ManagedCluster` CR with labels at the start and end of the GitOps ZTP process. When GitOps ZTP configuration postinstallation commences, the `ManagedCluster` has the `ztp-running` label applied. When all policies are remediated to the cluster and are fully compliant, these directives cause the TALM to remove the `ztp-running` label and apply the `ztp-done` label. For deployments that make use of the `informDuValidator` policy, the `ztp-done` label is applied when the cluster is fully ready for deployment of applications. This includes all reconciliation and resulting effects of the GitOps ZTP applied configuration CRs. The `ztp-done` label affects automatic `ClusterGroupUpgrade` CR creation by TALM. Do not manipulate this label after the initial GitOps ZTP installation of the cluster. Linked CRs : The automatically created `ClusterGroupUpgrade` CR has the owner reference set as the `ManagedCluster` from which it was derived. This reference ensures that deleting the `ManagedCluster` CR causes the instance of the `ClusterGroupUpgrade` to be deleted along with any supporting resources. ## Overview of deploying managed clusters with GitOps ZTP {#ztp-ztp-building-blocks_ztp-deploying-far-edge-sites} Red Hat Advanced Cluster Management (RHACM) uses GitOps Zero Touch Provisioning (ZTP) to deploy single-node OpenShift Container Platform clusters, three-node clusters, and standard clusters. You manage site configuration data as OpenShift Container Platform custom resources (CRs) in a Git repository. GitOps ZTP uses a declarative GitOps approach for a develop once, deploy anywhere model to deploy the managed clusters. The deployment of the clusters includes: - Installing the host operating system (RHCOS) on a blank server - Deploying OpenShift Container Platform - Creating cluster policies and site subscriptions - Making the necessary network configurations to the server operating system - Deploying profile Operators and performing any needed software-related configuration, such as performance profile, PTP, and SR-IOV > [!NOTE] > To deploy clusters with virtualized control planes running on OpenShift Virtualization VMs instead of physical servers, you can use KubeVirt Redfish to expose VMs as Redfish endpoints. For more information, see "Virtualized control planes". **Additional resources** {._additional-resources} - [Understanding virtualized control planes](/openshift-docs-markdown/vcp/vcp-overview#vcp-overview) ## Overview of the managed site installation process {#ztp-overview-managed-site-installation-process_ztp-deploying-far-edge-sites} After you apply the managed site custom resources (CRs) on the hub cluster, the following actions happen automatically: 1. A Discovery image ISO file is generated and booted on the target host. 2. When the ISO file successfully boots on the target host it reports the host hardware information to RHACM. 3. After all hosts are discovered, OpenShift Container Platform is installed. 4. When OpenShift Container Platform finishes installing, the hub installs the `klusterlet` service on the target cluster. 5. The requested add-on services are installed on the target cluster. The Discovery image ISO process is complete when the `Agent` CR for the managed cluster is created on the hub cluster. > [!IMPORTANT] > The target bare-metal host must meet the networking, firmware. For more information, see "Recommended single-node OpenShift cluster configuration for vDU application workloads". ## Creating the managed bare-metal host secrets {#ztp-creating-the-site-secrets_ztp-deploying-far-edge-sites} Add the required `Secret` custom resources (CRs) for the managed bare-metal host to the hub cluster. You need a secret for the GitOps Zero Touch Provisioning (ZTP) pipeline to access the Baseboard Management Controller (BMC) and a secret for the assisted installer service to pull cluster installation images from the registry. > [!NOTE] > The secrets are referenced from the `ClusterInstance` CR by name. The namespace must match the `ClusterInstance` namespace. **Procedure** 1. Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators: 1. Save the following YAML as the file `example-sno-secret.yaml`: ```yaml apiVersion: v1 kind: Secret metadata: name: example-sno-bmc-secret namespace: example-sno data: password: username: type: Opaque --- apiVersion: v1 kind: Secret metadata: name: pull-secret namespace: example-sno data: .dockerconfigjson: type: kubernetes.io/dockerconfigjson ``` where: `namespace` : Must match the namespace configured in the related `ClusterInstance` CR. `password`, `username` : Base64-encoded values for `password` and `username`. `.dockerconfigjson` : Base64-encoded pull secret. 2. Add the relative path to `example-sno-secret.yaml` to the `kustomization.yaml` file that you use to install the cluster. ## Configuring Discovery ISO kernel arguments for installations using GitOps ZTP {#setting-managed-bare-metal-host-kernel-arguments_ztp-deploying-far-edge-sites} The GitOps Zero Touch Provisioning (ZTP) workflow uses the Discovery ISO as part of the OpenShift Container Platform installation process on managed bare-metal hosts. You can edit the `InfraEnv` resource to specify kernel arguments for the Discovery ISO. This is useful for cluster installations with specific environmental requirements. For example, configure the `rd.net.timeout.carrier` kernel argument for the Discovery ISO to facilitate static networking for the cluster or to receive a DHCP address before downloading the root file system during installation. > [!NOTE] > In OpenShift Container Platform 4.22, you can only add kernel arguments. You can not replace or delete kernel arguments. **Prerequisites** - You have installed the OpenShift CLI (oc). - You have logged in to the hub cluster as a user with cluster-admin privileges. **Procedure** 1. Create the `InfraEnv` CR and edit the `spec.kernelArguments` specification to configure kernel arguments. 1. Save the following YAML in an `InfraEnv-example.yaml` file: > [!NOTE] > The `InfraEnv` CR in this example uses template syntax such as `{{ .Cluster.ClusterName }}` that is populated based on values in the `ClusterInstance` CR. The `ClusterInstance` CR automatically populates values for these templates during deployment. Do not edit the templates manually. ```yaml apiVersion: agent-install.openshift.io/v1beta1 kind: InfraEnv metadata: annotations: argocd.argoproj.io/sync-wave: "1" name: "{{ .Cluster.ClusterName }}" namespace: "{{ .Cluster.ClusterName }}" spec: clusterRef: name: "{{ .Cluster.ClusterName }}" namespace: "{{ .Cluster.ClusterName }}" kernelArguments: - operation: append value: audit=0 - operation: append value: trace=1 sshAuthorizedKey: "{{ .Site.SshPublicKey }}" proxy: "{{ .Cluster.ProxySettings }}" pullSecretRef: name: "{{ .Site.PullSecretRef.Name }}" ignitionConfigOverride: "{{ .Cluster.IgnitionConfigOverride }}" nmStateConfigLabelSelector: matchLabels: nmstate-label: "{{ .Cluster.ClusterName }}" additionalNTPSources: "{{ .Cluster.AdditionalNTPSources }}" ``` - `kernelArguments.operation` specifies the append operation to add a kernel argument. - `kernelArguments.value` specifies the kernel argument you want to configure. This example configures the `audit` kernel argument and the `trace` kernel argument. 2. Commit the `InfraEnv-example.yaml` file to your Git repository and push your changes. The following example shows a sample Git repository structure: ```text ~/example-ztp/install └── site-install ├── clusterinstance-example.yaml ├── InfraEnv-example.yaml └── kustomization.yaml ``` 3. Update the `kustomization.yaml` file to use the `configMapGenerator` field to package the `InfraEnv` CR into a `ConfigMap`: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - clusterinstance-example.yaml configMapGenerator: - name: custom-infraenv-cm namespace: example-cluster files: - InfraEnv-example.yaml generatorOptions: disableNameSuffixHash: true ``` - `clusterinstance-example.yaml` specifies the name of the `ClusterInstance` CR. - `configMapGenerator.name` specifies the name of the `ConfigMap` that contains the custom `InfraEnv` CR. - `configMapGenerator.namespace` must match the `ClusterInstance` namespace. 4. In your `ClusterInstance` CR, reference the `ConfigMap` in the `spec.templateRefs` field: ```yaml apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "example-cluster" namespace: "example-cluster" spec: clusterName: "example-cluster" templateRefs: - name: custom-infraenv-cm namespace: example-cluster # ... ``` - `spec.templateRefs` specifies the `ConfigMap` CR that contains the custom `InfraEnv` CR template. 5. Commit the `ClusterInstance` CR and `kustomization.yaml` to your Git repository and push your changes. When the Argo CD pipeline syncs the changes, the SiteConfig Operator uses the custom `InfraEnv-example` CR from the generated `ConfigMap` to configure the infrastructure environment, including the custom kernel arguments. **Verification** To verify that the kernel arguments are applied, after the Discovery image verifies that OpenShift Container Platform is ready for installation, you can SSH to the target host before the installation process begins. At that point, you can view the kernel arguments for the Discovery ISO in the `/proc/cmdline` file. 1. Begin an SSH session with the target host: ```terminal $ ssh -i /path/to/privatekey core@ ``` 2. View the system’s kernel arguments by using the following command: ```terminal $ cat /proc/cmdline ``` ## Deploying a managed cluster with ClusterInstance and GitOps ZTP {#ztp-deploying-a-site_ztp-deploying-far-edge-sites} Use the following procedure to create a `ClusterInstance` custom resource (CR) and related files and initiate the GitOps Zero Touch Provisioning (ZTP) cluster deployment. > [!NOTE] > You require Red Hat Advanced Cluster Management (RHACM) version 2.12 or later to install the SiteConfig Operator and use the `ClusterInstance` CR. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You installed the SiteConfig Operator in the hub cluster. - You have logged in to the hub cluster as a user with `cluster-admin` privileges. - You configured the hub cluster for generating the required installation and policy CRs. - You created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and you must configure it as a source repository for the ArgoCD application. See "Preparing the GitOps ZTP site configuration repository" for more information. > [!NOTE] > When you create the source repository, ensure that you patch the ArgoCD application with the `argocd/deployment/argocd-openshift-gitops-patch.json` patch-file that you extract from the `ztp-site-generate` container. See "Configuring the hub cluster with ArgoCD". - To be ready for provisioning managed clusters, you require the following for each bare-metal host: Network connectivity : Your network requires DNS. Managed cluster hosts should be reachable from the hub cluster. Ensure that Layer 3 connectivity exists between the hub cluster and the managed cluster host. Baseboard Management Controller (BMC) details : GitOps ZTP uses BMC username and password details to connect to the BMC during cluster installation. The GitOps ZTP plugin manages the `ManagedCluster` CRs on the hub cluster based on the `ClusterInstance` CR in your site Git repo. You create individual `BMCSecret` CRs for each host manually. **Procedure** 1. Create the required managed cluster secrets on the hub cluster. These resources must be in a namespace with a name matching the cluster name. For example, in `out/argocd/example/clusterinstance/example-sno.yaml`, the cluster name and namespace is `example-sno`. 1. Export the cluster namespace by running the following command: ```terminal $ export CLUSTERNS=example-sno ``` 2. Create the namespace: ```terminal $ oc create namespace $CLUSTERNS ``` 2. Create pull secret and BMC `Secret` CRs for the managed cluster. The pull secret must contain all the credentials necessary for installing OpenShift Container Platform and all required Operators. See "Creating the managed bare-metal host secrets" for more information. > [!NOTE] > The secrets are referenced from the `ClusterInstance` custom resource (CR) by name. The namespace must match the `ClusterInstance` namespace. 3. Create a `ClusterInstance` CR for your cluster in your local clone of the Git repository: 1. Choose the appropriate example for your CR from the `out/argocd/example/clusterinstance/` folder. The folder includes example files for single node, three-node, and standard clusters: - `example-sno.yaml` - `example-3node.yaml` - `example-standard.yaml` 2. Change the cluster and host details in the example file to match the type of cluster you want. For example: ```yaml {title="Example single-node OpenShift ClusterInstance CR"} # example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-ai-sno --- apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "example-ai-sno" namespace: "example-ai-sno" spec: baseDomain: "example.com" pullSecretRef: name: "assisted-deployment-pull-secret" clusterImageSetNameRef: "openshift-4.22" sshPublicKey: "ssh-rsa AAAA..." clusterName: "example-ai-sno" networkType: "OVNKubernetes" # installConfigOverrides is a generic way of passing install-config # parameters through the siteConfig. The 'capabilities' field configures # the composable openshift feature. In this 'capabilities' setting, we # remove all the optional set of components. # Notes: # - OperatorLifecycleManager is needed for 4.15 and later # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier # - Ingress is needed for 4.16 and later installConfigOverrides: | { "capabilities": { "baselineCapabilitySet": "None", "additionalEnabledCapabilities": [ "NodeTuning", "OperatorLifecycleManager", "Ingress" ] } } # Include references to extraManifest ConfigMaps. extraManifestsRefs: - name: sno-extra-manifest-configmap extraLabels: ManagedCluster: # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples du-profile: "latest" # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates: # ../policygentemplates/common-ranGen.yaml will apply to all clusters with 'common: true' common: "true" # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""' group-du-sno: "" # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"' # Normally this should match or contain the cluster name so it only applies to a single cluster sites : "example-sno" clusterNetwork: - cidr: 1001:1::/48 hostPrefix: 64 machineNetwork: - cidr: 1111:2222:3333:4444::/64 serviceNetwork: - cidr: 1001:2::/112 additionalNTPSources: - 1111:2222:3333:4444::2 # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate # please see Workload Partitioning Feature for a complete guide. cpuPartitioningMode: AllNodes templateRefs: - name: ai-cluster-templates-v1 namespace: open-cluster-management nodes: - hostName: "example-node1.example.com" role: "master" bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1" bmcCredentialsName: name: "example-node1-bmh-secret" bootMACAddress: "AA:BB:CC:DD:EE:11" # Use UEFISecureBoot to enable secure boot, UEFI to disable. bootMode: "UEFISecureBoot" rootDeviceHints: deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0" # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md in argocd folder for more details ignitionConfigOverride: | { "ignition": { "version": "3.2.0" }, "storage": { "disks": [ { "device": "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0", "partitions": [ { "label": "var-lib-containers", "sizeMiB": 0, "startMiB": 250000 } ], "wipeTable": false } ], "filesystems": [ { "device": "/dev/disk/by-partlabel/var-lib-containers", "format": "xfs", "mountOptions": [ "defaults", "prjquota" ], "path": "/var/lib/containers", "wipeFilesystem": true } ] }, "systemd": { "units": [ { "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target", "enabled": true, "name": "var-lib-containers.mount" } ] } } nodeNetwork: interfaces: - name: eno1 macAddress: "AA:BB:CC:DD:EE:11" config: interfaces: - name: eno1 type: ethernet state: up ipv4: enabled: false ipv6: enabled: true address: # For SNO sites with static IP addresses, the node-specific, # API and Ingress IPs should all be the same and configured on # the interface - ip: 1111:2222:3333:4444::aaaa:1 prefix-length: 64 dns-resolver: config: search: - example.com server: - 1111:2222:3333:4444::2 routes: config: - destination: ::/0 next-hop-interface: eno1 next-hop-address: 1111:2222:3333:4444::1 table-id: 254 templateRefs: - name: ai-node-templates-v1 namespace: open-cluster-management ``` > [!NOTE] > For more information about BMC addressing, see the "Additional resources" section. The `installConfigOverrides` and `ignitionConfigOverride` fields are expanded in the example for ease of readability. > [!NOTE] > To override the default `BareMetalHost` CR for a node, create a custom node template in a `ConfigMap` and reference it in the node-level `spec.nodes.templateRefs` field in the `ClusterInstance` CR. Ensure that you set the `argocd.argoproj.io/sync-wave: "3"` annotation in your override `BareMetalHost` CR. 3. You can inspect the default set of extra-manifest `MachineConfig` CRs in `out/argocd/extra-manifest`. It is automatically applied to the cluster when it is installed. 4. Optional: To provision additional install-time manifests on the provisioned cluster, package your extra manifest CRs in a `ConfigMap` and reference it in the `extraManifestsRefs` field of the `ClusterInstance` CR. For more information, see "Customizing extra installation manifests in the GitOps ZTP pipeline". > [!IMPORTANT] > For optimal cluster performance, enable crun for master and worker nodes in single-node OpenShift, single-node OpenShift with additional worker nodes, three-node OpenShift, and standard clusters. > > Enable crun in a `ContainerRuntimeConfig` CR as an additional Day 0 install-time manifest to avoid the cluster having to reboot. > > The `enable-crun-master.yaml` and `enable-crun-worker.yaml` CR files are in the `out/source-crs/optional-extra-manifest/` folder that you can extract from the `ztp-site-generate` container. 4. Add the `ClusterInstance` CR to the `kustomization.yaml` file in the `generators` section, similar to the example shown in `out/argocd/example/clusterinstance/kustomization.yaml`. 5. Commit the `ClusterInstance` CR and associated `kustomization.yaml` changes in your Git repository and push the changes. The ArgoCD pipeline detects the changes and begins the managed cluster deployment. **Verification** - Verify that the custom roles and labels are applied after the node is deployed: ```terminal $ oc describe node example-node.example.com ``` The following example output shows the custom roles and labels: ```terminal Name: example-node.example.com Roles: control-plane,example-label,master,worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux custom-label/parameter1=true kubernetes.io/arch=amd64 kubernetes.io/hostname=cnfdf03.telco5gran.eng.rdu2.redhat.com kubernetes.io/os=linux node-role.kubernetes.io/control-plane= node-role.kubernetes.io/example-label= node-role.kubernetes.io/master= node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos ``` - `node-role.kubernetes.io/example-label=` shows the custom label applied to the node. **Additional resources** {._additional-resources} - [Single-node OpenShift ClusterInstance CR installation reference](/openshift-docs-markdown/edge_computing/ztp-deploying-far-edge-sites#ztp-clusterinstance-config-reference_ztp-deploying-far-edge-sites) ### Configuring IPsec encryption for single-node OpenShift clusters using GitOps ZTP and ClusterInstance resources {#ztp-configuring-ipsec-using-ztp-and-siteconfig_ztp-deploying-far-edge-sites} You can enable IPsec encryption in managed single-node OpenShift clusters that you install using GitOps ZTP and Red Hat Advanced Cluster Management (RHACM). You can encrypt traffic between the managed cluster and IPsec endpoints external to the managed cluster. All network traffic between nodes on the OVN-Kubernetes cluster network is encrypted with IPsec in Transport mode. > [!IMPORTANT] > You can also configure IPsec encryption for single-node OpenShift clusters with an additional worker node by following this procedure. It is recommended to use the `MachineConfig` custom resource (CR) to configure IPsec encryption for single-node OpenShift clusters and single-node OpenShift clusters with an additional worker node because of their low resource availability. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. - You have installed the SiteConfig Operator in the hub cluster. - You have configured RHACM and the hub cluster for generating the required installation and policy custom resources (CRs) for managed clusters. - You have created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and be defined as a source repository for the Argo CD application. - You have installed the `butane` utility version 0.20.0 or later. - You have a PKCS#12 certificate for the IPsec endpoint and a CA cert in PEM format. **Procedure** 1. Extract the latest version of the `ztp-site-generate` container source and merge it with your repository where you manage your custom site configuration data. 2. Configure `optional-extra-manifest/ipsec/ipsec-endpoint-config.yaml` with the required values that configure IPsec in the cluster. For example: ```yaml interfaces: - name: hosta_conn type: ipsec libreswan: left: '%defaultroute' leftid: '%fromcert' leftmodecfgclient: false leftcert: left_server leftrsasigkey: '%cert' right: rightid: '%fromcert' rightrsasigkey: '%cert' rightsubnet: ikev2: insist type: tunnel ``` - `leftcert` must match the name of the certificate used on the remote system. - `right` is the external host IP address or DNS hostname. - `rightsubnet` is the IP subnet of the external host on the other side of the IPsec tunnel. - `ikev2: insist` uses the IKEv2 VPN encryption protocol only. Do not use IKEv1, which is deprecated. 3. Add the following certificates to the `optional-extra-manifest/ipsec` folder: - `left_server.p12`: The certificate bundle for the IPsec endpoints - `ca.pem`: The certificate authority that you signed your certificates with The certificate files are required for the Network Security Services (NSS) database on each host. These files are imported as part of the Butane configuration in later steps. 4. Open a shell prompt at the `optional-extra-manifest/ipsec` folder of the Git repository where you maintain your custom site configuration data. 5. Run the `optional-extra-manifest/ipsec/build.sh` script to generate the required Butane and `MachineConfig` CRs files. If the PKCS#12 certificate is protected with a password, set the `-W` argument. The following example shows the generated output directory structure: ```terminal out └── argocd └── example └── optional-extra-manifest └── ipsec ├── 99-ipsec-master-endpoint-config.bu ├── 99-ipsec-master-endpoint-config.yaml ├── 99-ipsec-worker-endpoint-config.bu ├── 99-ipsec-worker-endpoint-config.yaml ├── build.sh ├── ca.pem ├── left_server.p12 ├── enable-ipsec.yaml ├── ipsec-endpoint-config.yml └── README.md ``` - The `ipsec/build.sh` script generates the Butane and endpoint configuration CRs. - Add the `ca.pem` and `left_server.p12` certificate files that are relevant to your network. 6. Create an `ipsec-manifests/` folder in the repository where you manage your custom site configuration data. Add the `enable-ipsec.yaml` and `99-ipsec-*` YAML files to the directory. For example: ```terminal site-configs/ ├── hub-1/ │ └── clusterinstance-site1-sno-du.yaml ├── ipsec-manifests/ │ ├── enable-ipsec.yaml │ ├── 99-ipsec-worker-endpoint-config.yaml │ └── 99-ipsec-master-endpoint-config.yaml └── kustomization.yaml ``` 7. Create a `kustomization.yaml` file that uses `configMapGenerator` to package your IPsec manifests into a `ConfigMap`: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - hub-1/clusterinstance-site1-sno-du.yaml configMapGenerator: - name: ipsec-manifests-cm namespace: site1-sno-du files: - ipsec-manifests/enable-ipsec.yaml - ipsec-manifests/99-ipsec-master-endpoint-config.yaml - ipsec-manifests/99-ipsec-worker-endpoint-config.yaml generatorOptions: disableNameSuffixHash: true ``` - `namespace` must match the `ClusterInstance` namespace. - `disableNameSuffixHash: true` disables the hash suffix so the `ConfigMap` name is predictable. 8. In your `ClusterInstance` CR, reference the `ConfigMap` in the `extraManifestsRefs` field: ```yaml apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "site1-sno-du" namespace: "site1-sno-du" spec: clusterName: "site1-sno-du" networkType: "OVNKubernetes" extraManifestsRefs: - name: ipsec-manifests-cm # ... ``` - `extraManifestsRefs.name` references the `ConfigMap` containing the IPsec manifests. > [!NOTE] > If you have other extra manifests, you can either include them in the same `ConfigMap` or create multiple `ConfigMap` resources and reference each of those in the `extraManifestsRefs` field. 9. Commit the `ClusterInstance` CR, IPsec manifest files, and `kustomization.yaml` changes in your Git repository and push the changes to provision the managed cluster and configure IPsec encryption. The Argo CD pipeline detects the changes and begins the managed cluster deployment. During cluster provisioning, the SiteConfig Operator applies the CRs contained in the referenced `ConfigMap` resources as extra manifests. **Verification** For information about verifying the IPsec encryption, see "Verifying the IPsec encryption". **Additional resources** {._additional-resources} - [Verifying the IPsec encryption](/openshift-docs-markdown/edge_computing/ztp-deploying-far-edge-sites#ztp-verifying-ipsec_ztp-deploying-far-edge-sites) - [Configuring IPsec encryption](/openshift-docs-markdown/networking/network_security/configuring-ipsec-ovn#configuring-ipsec-ovn) - [Encryption protocol and IPsec mode](/openshift-docs-markdown/networking/network_security/configuring-ipsec-ovn#nw-ovn-ipsec-encryption_configuring-ipsec-ovn) - [Installing managed clusters with RHACM and ClusterInstance resources](/openshift-docs-markdown/edge_computing/ztp-deploying-far-edge-sites#ztp-deploying-far-edge-sites) ### Configuring IPsec encryption for multi-node clusters using GitOps ZTP and ClusterInstance resources {#ztp-configuring-ipsec-using-ztp-and-siteconfig-for-mno_ztp-deploying-far-edge-sites} You can enable IPsec encryption in managed multi-node clusters that you install using GitOps ZTP and Red Hat Advanced Cluster Management (RHACM). You can encrypt traffic between the managed cluster and IPsec endpoints external to the managed cluster. All network traffic between nodes on the OVN-Kubernetes cluster network is encrypted with IPsec in Transport mode. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. - You have installed the SiteConfig Operator in the hub cluster. - You have configured RHACM and the hub cluster for generating the required installation and policy custom resources (CRs) for managed clusters. - You have created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and be defined as a source repository for the Argo CD application. - You have installed the `butane` utility version 0.20.0 or later. - You have a PKCS#12 certificate for the IPsec endpoint and a CA cert in PEM format. - You have installed the NMState Operator. **Procedure** 1. Extract the latest version of the `ztp-site-generate` container source and merge it with your repository where you manage your custom site configuration data. 2. Configure the `optional-extra-manifest/ipsec/ipsec-config-policy.yaml` file with the required values that configure IPsec in the cluster. ```yaml {title="ConfigurationPolicy object for creating an IPsec configuration"} apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-config spec: namespaceSelector: include: ["default"] exclude: [] matchExpressions: [] matchLabels: {} remediationAction: inform severity: low evaluationInterval: compliant: noncompliant: object-templates-raw: | {{- range (lookup "v1" "Node" "" "").items }} - complianceType: musthave objectDefinition: kind: NodeNetworkConfigurationPolicy apiVersion: nmstate.io/v1 metadata: name: {{ .metadata.name }}-ipsec-policy spec: nodeSelector: kubernetes.io/hostname: {{ .metadata.name }} desiredState: interfaces: - name: hosta_conn type: ipsec libreswan: left: '%defaultroute' leftid: '%fromcert' leftmodecfgclient: false leftcert: left_server leftrsasigkey: '%cert' right: rightid: '%fromcert' rightrsasigkey: '%cert' rightsubnet: ikev2: insist type: tunnel ``` - `leftcert` must match the name of the certificate used on the remote system. - `right` is the external host IP address or DNS hostname. - `rightsubnet` is the IP subnet of the external host on the other side of the IPsec tunnel. - `ikev2: insist` uses the IKEv2 VPN encryption protocol only. Do not use IKEv1, which is deprecated. 3. Add the following certificates to the `optional-extra-manifest/ipsec` folder: - `left_server.p12`: The certificate bundle for the IPsec endpoints - `ca.pem`: The certificate authority that you signed your certificates with The certificate files are required for the Network Security Services (NSS) database on each host. These files are imported as part of the Butane configuration in later steps. 4. Open a shell prompt at the `optional-extra-manifest/ipsec` folder of the Git repository where you maintain your custom site configuration data. 5. Run the `optional-extra-manifest/ipsec/import-certs.sh` script to generate the required Butane and `MachineConfig` CRs to import the external certs. If the PKCS#12 certificate is protected with a password, set the `-W` argument. The following example shows the generated output directory structure: ```terminal out └── argocd └── example └── optional-extra-manifest └── ipsec ├── 99-ipsec-master-import-certs.bu ├── 99-ipsec-master-import-certs.yaml ├── 99-ipsec-worker-import-certs.bu ├── 99-ipsec-worker-import-certs.yaml ├── import-certs.sh ├── ca.pem ├── left_server.p12 ├── enable-ipsec.yaml ├── ipsec-config-policy.yaml └── README.md ``` - The `ipsec/import-certs.sh` script generates the Butane and endpoint configuration CRs. - Add the `ca.pem` and `left_server.p12` certificate files that are relevant to your network. 6. Create an `ipsec-manifests/` folder in the repository where you manage your custom site configuration data and add the `enable-ipsec.yaml` and `99-ipsec-*` YAML files to the directory. ```terminal {title="Example site configuration directory"} site-configs/ ├── hub-1/ │ └── clusterinstance-site1-mno-du.yaml ├── ipsec-manifests/ │ ├── enable-ipsec.yaml │ ├── 99-ipsec-master-import-certs.yaml │ └── 99-ipsec-worker-import-certs.yaml └── kustomization.yaml ``` 7. Create a `kustomization.yaml` file that uses `configMapGenerator` to package your IPsec manifests into a `ConfigMap`: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - hub-1/clusterinstance-site1-mno-du.yaml configMapGenerator: - name: ipsec-manifests-cm namespace: site1-mno-du files: - ipsec-manifests/enable-ipsec.yaml - ipsec-manifests/99-ipsec-master-import-certs.yaml - ipsec-manifests/99-ipsec-worker-import-certs.yaml generatorOptions: disableNameSuffixHash: true ``` - `namespace` must match the `ClusterInstance` namespace. - `disableNameSuffixHash: true` disables the hash suffix so the `ConfigMap` name is predictable. 8. In your `ClusterInstance` CR, reference the `ConfigMap` in the `extraManifestsRefs` field: ```yaml apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "site1-mno-du" namespace: "site1-mno-du" spec: clusterName: "site1-mno-du" networkType: "OVNKubernetes" extraManifestsRefs: - name: ipsec-manifests-cm # ... ``` - `extraManifestsRefs.name` references the `ConfigMap` containing the IPsec certificate import manifests. > [!NOTE] > If you have other extra manifests, you can either include them in the same `ConfigMap` or create multiple `ConfigMap` resources and reference them all in `extraManifestsRefs`. 9. Include the `ipsec-config-policy.yaml` config policy file in the `source-crs` directory in GitOps and reference the file in one of the `PolicyGenerator` CRs. 10. Commit the `ClusterInstance` CR, IPsec manifest files, and `kustomization.yaml` changes in your Git repository and push the changes to provision the managed cluster and configure IPsec encryption. The Argo CD pipeline detects the changes and begins the managed cluster deployment. During cluster provisioning, the SiteConfig Operator applies the CRs contained in the referenced `ConfigMap` resources as extra manifests. The IPsec configuration policy is applied as a Day 2 operation after the cluster is provisioned. **Verification** For information about verifying the IPsec encryption, see "Verifying the IPsec encryption". **Additional resources** {._additional-resources} - [Verifying the IPsec encryption](/openshift-docs-markdown/edge_computing/ztp-deploying-far-edge-sites#ztp-verifying-ipsec_ztp-deploying-far-edge-sites) - [Configuring IPsec encryption](/openshift-docs-markdown/networking/network_security/configuring-ipsec-ovn#configuring-ipsec-ovn) - [Encryption protocol and IPsec mode](/openshift-docs-markdown/networking/network_security/configuring-ipsec-ovn#nw-ovn-ipsec-encryption_configuring-ipsec-ovn) - [Installing managed clusters with RHACM and ClusterInstance resources](/openshift-docs-markdown/edge_computing/ztp-deploying-far-edge-sites#ztp-deploying-far-edge-sites) ### Verifying the IPsec encryption {#ztp-verifying-ipsec_ztp-deploying-far-edge-sites} You can verify that the IPsec encryption is successfully applied in a managed OpenShift Container Platform cluster. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. - You have configured the IPsec encryption. **Procedure** 1. Start a debug pod for the managed cluster by running the following command: ```terminal $ oc debug node/ ``` 2. Check that the IPsec policy is applied in the cluster node by running the following command: ```terminal sh-5.1# ip xfrm policy ``` The following example output shows the IPsec policy applied to the cluster node: ```terminal src 172.16.123.0/24 dst 10.1.232.10/32 dir out priority 1757377 ptype main tmpl src 10.1.28.190 dst 10.1.232.10 proto esp reqid 16393 mode tunnel src 10.1.232.10/32 dst 172.16.123.0/24 dir fwd priority 1757377 ptype main tmpl src 10.1.232.10 dst 10.1.28.190 proto esp reqid 16393 mode tunnel src 10.1.232.10/32 dst 172.16.123.0/24 dir in priority 1757377 ptype main tmpl src 10.1.232.10 dst 10.1.28.190 proto esp reqid 16393 mode tunnel ``` 3. Check that the IPsec tunnel is up and connected by running the following command: ```terminal sh-5.1# ip xfrm state ``` The following example output shows the IPsec tunnel is up and connected: ```terminal src 10.1.232.10 dst 10.1.28.190 proto esp spi 0xa62a05aa reqid 16393 mode tunnel replay-window 0 flag af-unspec esn auth-trunc hmac(sha1) 0x8c59f680c8ea1e667b665d8424e2ab749cec12dc 96 enc cbc(aes) 0x2818a489fe84929c8ab72907e9ce2f0eac6f16f2258bd22240f4087e0326badb anti-replay esn context: seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0 replay_window 128, bitmap-length 4 00000000 00000000 00000000 00000000 src 10.1.28.190 dst 10.1.232.10 proto esp spi 0x8e96e9f9 reqid 16393 mode tunnel replay-window 0 flag af-unspec esn auth-trunc hmac(sha1) 0xd960ddc0a6baaccb343396a51295e08cfd8aaddd 96 enc cbc(aes) 0x0273c02e05b4216d5e652de3fc9b3528fea94648bc2b88fa01139fdf0beb27ab anti-replay esn context: seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0 replay_window 128, bitmap-length 4 00000000 00000000 00000000 00000000 ``` 4. Ping a known IP in the external host subnet by running the following command: For example, ping an IP address in the `rightsubnet` range that you set in the `ipsec/ipsec-endpoint-config.yaml` file: ```terminal sh-5.1# ping 172.16.110.8 ``` The following example output shows a successful ping response: ```terminal PING 172.16.110.8 (172.16.110.8) 56(84) bytes of data. 64 bytes from 172.16.110.8: icmp_seq=1 ttl=64 time=153 ms 64 bytes from 172.16.110.8: icmp_seq=2 ttl=64 time=155 ms ``` ### ClusterInstance CR installation reference {#ztp-clusterinstance-config-reference_ztp-deploying-far-edge-sites} For a detailed API reference for the `ClusterInstance` custom resource, see [ClusterInstance API](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/apis/index#clusterinstance-api) in the Red Hat Advanced Cluster Management (RHACM) documentation. **Additional resources** {._additional-resources} - [Customizing extra installation manifests in the GitOps ZTP pipeline](/openshift-docs-markdown/edge_computing/ztp-advanced-install-ztp#ztp-customizing-the-install-extra-manifests_ztp-advanced-install-ztp) - [Preparing the GitOps ZTP site configuration repository](/openshift-docs-markdown/edge_computing/ztp-preparing-the-hub-cluster#ztp-preparing-the-ztp-git-repository_ztp-preparing-the-hub-cluster) - [Configuring the hub cluster with ArgoCD](/openshift-docs-markdown/edge_computing/ztp-preparing-the-hub-cluster#ztp-configuring-hub-cluster-with-argocd_ztp-preparing-the-hub-cluster) - [Signalling GitOps ZTP cluster deployment completion with validator inform policies](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-advanced-policygenerator-config#ztp-creating-a-validator-inform-policy_ztp-advanced-policy-config) - [Creating the managed bare-metal host secrets](/openshift-docs-markdown/edge_computing/ztp-manual-install#ztp-creating-the-site-secrets_ztp-manual-install) - [BMC addressing](/openshift-docs-markdown/installing/installing_bare_metal/ipi/ipi-install-installation-workflow#bmc-addressing_ipi-install-installation-workflow) - [About root device hints](/openshift-docs-markdown/installing/installing_with_agent_based_installer/preparing-to-install-with-agent-based-installer#root-device-hints_preparing-to-install-with-agent-based-installer) ## Managing host firmware settings with GitOps ZTP {#ztp-configuring-host-firmware-with-gitops-ztp_ztp-deploying-far-edge-sites} Hosts require the correct firmware configuration to ensure high performance and optimal efficiency. You can deploy custom host firmware configurations for managed clusters with GitOps ZTP. Tune hosts with specific hardware profiles in your lab and ensure they are optimized for your requirements. When you have completed host tuning to your satisfaction, you extract the host profile and save it in your GitOps ZTP repository. Then, you use the host profile to configure firmware settings in the managed cluster hosts that you deploy with GitOps ZTP. You specify the required hardware profiles by creating `HostFirmwareSettings` CRs, packaging them in `ConfigMap` resources, and referencing them in the `templateRefs` field of your `ClusterInstance` CR. The SiteConfig Operator generates the required `HostFirmwareSettings` and `BareMetalHost` CRs that are applied to the hub cluster. Use the following best practices to manage your host firmware profiles. Identify critical firmware settings with hardware vendors : Work with hardware vendors to identify and document critical host firmware settings required for optimal performance and compatibility with the deployed host platform. Use common firmware configurations across similar hardware platforms : Where possible, use a standardized host firmware configuration across similar hardware platforms to reduce complexity and potential errors during deployment. Test firmware configurations in a lab environment : Test host firmware configurations in a controlled lab environment before deploying in production to ensure that settings are compatible with hardware, firmware, and software. Manage firmware profiles in source control : Manage host firmware profiles in Git repositories to track changes, ensure consistency, and facilitate collaboration with vendors. **Additional resources** {._additional-resources} - [Recommended firmware configuration for vDU cluster hosts](/openshift-docs-markdown/edge_computing/ztp-vdu-validating-cluster-tuning#ztp-du-firmware-config-reference_vdu-config-ref) ### Retrieving the host firmware schema for a managed cluster {#ztp-retrieving-the-host-firmware-schema_ztp-deploying-far-edge-sites} You can discover the host firmware schema for managed clusters. The host firmware schema for bare-metal hosts is populated with information that the Ironic API returns. The API returns information about host firmware interfaces, including firmware setting types, allowable values, ranges, and flags. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have installed Red Hat Advanced Cluster Management (RHACM) and logged in to the hub cluster as a user with `cluster-admin` privileges. - You have provisioned a cluster that is managed by RHACM. **Procedure** - Discover the host firmware schema for the managed cluster. Run the following command: ```terminal $ oc get firmwareschema -n -o yaml ``` The following example output shows the host firmware schema: ```terminal apiVersion: v1 items: - apiVersion: metal3.io/v1alpha1 kind: FirmwareSchema metadata: creationTimestamp: "2024-09-11T10:29:43Z" generation: 1 name: schema-40562318 namespace: compute-1 ownerReferences: - apiVersion: metal3.io/v1alpha1 kind: HostFirmwareSettings name: compute-1.example.com uid: 65d0e89b-1cd8-4317-966d-2fbbbe033fe9 resourceVersion: "280057624" uid: 511ad25d-f1c9-457b-9a96-776605c7b887 spec: schema: AccessControlService: allowable_values: - Enabled - Disabled attribute_type: Enumeration read_only: false # ... ``` ### Retrieving the host firmware settings for a managed cluster {#ztp-retrieving-the-host-firmware-settings_ztp-deploying-far-edge-sites} You can retrieve the host firmware settings for managed clusters. This is useful when you have deployed changes to the host firmware and you want to monitor the changes and ensure that they are applied successfully. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have installed Red Hat Advanced Cluster Management (RHACM) and logged in to the hub cluster as a user with `cluster-admin` privileges. - You have provisioned a cluster that is managed by RHACM. **Procedure** 1. Retrieve the host firmware settings for the managed cluster. Run the following command: ```terminal $ oc get hostfirmwaresettings -n -o yaml ``` The following example output shows the host firmware settings: ```terminal apiVersion: v1 items: - apiVersion: metal3.io/v1alpha1 kind: HostFirmwareSettings metadata: creationTimestamp: "2024-09-11T10:29:43Z" generation: 1 name: compute-1.example.com namespace: kni-qe-24 ownerReferences: - apiVersion: metal3.io/v1alpha1 blockOwnerDeletion: true controller: true kind: BareMetalHost name: compute-1.example.com uid: 0baddbb7-bb34-4224-8427-3d01d91c9287 resourceVersion: "280057626" uid: 65d0e89b-1cd8-4317-966d-2fbbbe033fe9 spec: settings: {} status: conditions: - lastTransitionTime: "2024-09-11T10:29:43Z" message: "" observedGeneration: 1 reason: Success # Indicates that a change in the host firmware settings has been detected. status: "True" type: ChangeDetected - lastTransitionTime: "2024-09-11T10:29:43Z" message: Invalid BIOS setting observedGeneration: 1 reason: ConfigurationError # Indicates that the host has an invalid firmware setting. status: "False" type: Valid lastUpdated: "2024-09-11T10:29:43Z" schema: name: schema-40562318 namespace: compute-1 # Contains the complete list of configured host firmware settings returned under the `status.settings` field. settings: AccessControlService: Enabled AcpiHpet: Enabled AcpiRootBridgePxm: Enabled # ... ``` 2. Optional: Check the status of the `HostFirmwareSettings` (`hfs`) custom resource in the cluster: ```terminal $ oc get hfs -n -o jsonpath='{.status.conditions[?(@.type=="ChangeDetected")].status}' ``` The following example output shows a detected change: ```terminal True ``` 3. Optional: Check for invalid firmware settings in the cluster host by running the following command: ```terminal $ oc get hfs -n -o jsonpath='{.status.conditions[?(@.type=="Valid")].status}' ``` The following example output shows an invalid firmware setting: ```terminal False ``` ### Deploying user-defined firmware to cluster hosts with GitOps ZTP {#ztp-deploying-user-defined-firmware-configuration-with-gitops-ztp_ztp-deploying-far-edge-sites} You can deploy user-defined firmware settings to cluster hosts by creating custom node templates that include `HostFirmwareSettings` CRs, and referencing them in the `ClusterInstance` CR. You can configure hardware profiles to apply to hosts in the following scenarios: - All hosts in the cluster - Individual hosts in the cluster > [!IMPORTANT] > You can configure host hardware profiles to be applied in a hierarchy. Node-level profiles override cluster-wide settings. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have installed Red Hat Advanced Cluster Management (RHACM) version 2.12 or later and logged in to the hub cluster as a user with `cluster-admin` privileges. - You have installed the SiteConfig Operator in the hub cluster. - You created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and be defined as a source repository for the Argo CD application. **Procedure** 1. Create the `HostFirmwareSettings` CR that contains the firmware settings you want to apply. For example, create the following YAML file: ```yaml {title="host-firmware-settings.yaml"} apiVersion: metal3.io/v1alpha1 kind: HostFirmwareSettings metadata: name: "site1-sno-du" namespace: "site1-sno-du" spec: settings: BootMode: "Uefi" LogicalProc: "Enabled" ProcVirtualization: "Enabled" ``` 2. Save the `HostFirmwareSettings` CR file relative to the `kustomization.yaml` file that you use to provision the cluster. For example: ```terminal site-configs/ └── site1-sno-du/ ├── clusterinstance-site1-sno-du.yaml ├── kustomization.yaml └── host-firmware-settings.yaml ``` 3. Create a `ConfigMap` to store the `HostFirmwareSettings` CR. You can use a `kustomization.yaml` file with `configMapGenerator` to create the `ConfigMap`. For example: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - clusterinstance-site1-sno-du.yaml configMapGenerator: - name: host-firmware-settings-cm namespace: site1-sno-du files: - host-firmware-settings.yaml generatorOptions: disableNameSuffixHash: true ``` - `namespace` must match the `ClusterInstance` namespace. - `host-firmware-settings.yaml` is the name of the `HostFirmwareSettings` CR. 4. To apply a hardware profile to all hosts in the cluster, reference the `ConfigMap` in the `spec.templateRefs` field of your `ClusterInstance` CR. For example: ```yaml apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "site1-sno-du" namespace: "site1-sno-du" spec: clusterName: "site1-sno-du" # ... templateRefs: - name: host-firmware-settings-cm namespace: site1-sno-du nodes: - hostName: "node1.example.com" # ... ``` - `templateRefs` applies the firmware profile to all hosts in the cluster. 5. Optional: To apply a hardware profile to a specific host in the cluster, reference the `ConfigMap` in the `spec.nodes[].templateRefs` field. For example: ```yaml apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "site1-sno-du" namespace: "site1-sno-du" spec: clusterName: "site1-sno-du" # ... nodes: - hostName: "node1.example.com" # ... templateRefs: - name: host-firmware-node1-cm namespace: site1-sno-du - hostName: "node2.example.com" # ... ``` - `nodes[].templateRefs` applies the firmware profile only to the `node1.example.com` host. > [!NOTE] > Node-level `templateRefs` settings override cluster-level `templateRefs` settings. 6. Commit the `ClusterInstance` CR, `ConfigMap`, and associated `kustomization.yaml` changes in your Git repository and push the changes. The Argo CD pipeline detects the changes and begins the managed cluster deployment. > [!NOTE] > Cluster deployment proceeds even if an invalid firmware setting is detected. To apply a correction using GitOps ZTP, re-deploy the cluster with the corrected hardware profile. **Verification** - Check that the firmware settings have been applied in the managed cluster host. For example, run the following command: ```terminal $ oc get hfs -n -o jsonpath='{.status.conditions[?(@.type=="Valid")].status}' ``` - `` is the namespace of the managed cluster and `` is the name of the managed cluster. The following example output shows valid firmware settings: ```terminal True ``` ## Monitoring managed cluster installation progress {#ztp-monitoring-deployment-progress_ztp-deploying-far-edge-sites} The Argo CD pipeline syncs the `ClusterInstance` CR from the Git repository to the hub cluster. The SiteConfig Operator then processes the `ClusterInstance` CR and generates the required cluster configuration CRs. You can monitor the progress of the cluster installation from the RHACM dashboard or from the command line. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. **Procedure** 1. Monitor the progress of cluster installation by running the following commands: 1. Export the cluster name: ```terminal $ export CLUSTER= ``` 2. Query the `AgentClusterInstall` CR for the managed cluster: ```terminal $ oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Completed")]}' | jq ``` 3. Get the installation events for the cluster: ```terminal $ curl -sk $(oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.debugInfo.eventsURL}') | jq '.[-2,-1]' ``` ## Troubleshooting GitOps ZTP by validating the installation CRs {#ztp-troubleshooting-ztp-gitops-installation-crs_ztp-deploying-far-edge-sites} The ArgoCD pipeline uses the `ClusterInstance` and `PolicyGenerator` or `PolicyGentemplate` custom resources (CRs) to generate the cluster configuration CRs and Red Hat Advanced Cluster Management (RHACM) policies. Use the following steps to troubleshoot issues that might occur during this process. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. **Procedure** 1. Check that the installation CRs were created by using the following command: ```terminal $ oc get AgentClusterInstall -n ``` If no object is returned, use the following steps to troubleshoot the ArgoCD pipeline flow from `ClusterInstance` files to the installation CRs. 2. Verify that the `ManagedCluster` CR was generated using the `ClusterInstance` CR on the hub cluster: ```terminal $ oc get managedcluster ``` 3. If the `ManagedCluster` is missing, check if the `clusters` application failed to synchronize the files from the Git repository to the hub cluster: ```terminal $ oc get applications.argoproj.io -n openshift-gitops clusters -o yaml ``` ## Troubleshooting GitOps ZTP virtual media booting on SuperMicro servers {#ztp-troubleshooting-ztp-gitops-supermicro-tls_ztp-deploying-far-edge-sites} SuperMicro X11 servers do not support virtual media installations when the image is served using the `https` protocol. As a result, single-node OpenShift deployments for this environment fail to boot on the target node. To avoid this issue, log in to the hub cluster and disable Transport Layer Security (TLS) in the `Provisioning` resource. This ensures the image is not served with TLS even though the image address uses the `https` scheme. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. **Procedure** 1. Disable TLS in the `Provisioning` resource by running the following command: ```terminal $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"disableVirtualMediaTLS": true}}' ``` 2. Continue the steps to deploy your single-node OpenShift cluster. ## Removing a managed cluster site from the GitOps ZTP pipeline {#ztp-site-cleanup_ztp-deploying-far-edge-sites} You can remove a managed site and the associated installation and configuration policy CRs from the GitOps Zero Touch Provisioning (ZTP) pipeline. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. **Procedure** 1. Remove a site and the associated CRs by removing the associated `ClusterInstance` and `PolicyGenerator` or `PolicyGentemplate` files from the `kustomization.yaml` file. 2. Add the following `syncOptions` field to the ArgoCD application that manages the target site. ```yaml kind: Application spec: syncPolicy: syncOptions: - PrunePropagationPolicy=background ``` When you run the GitOps ZTP pipeline again, the generated CRs are removed. 3. Optional: If you want to permanently remove a site, you should also remove the `ClusterInstance` and site-specific `PolicyGenerator` or `PolicyGentemplate` files from the Git repository. 4. Optional: If you want to remove a site temporarily, for example when redeploying a site, you can leave the `ClusterInstance` and site-specific `PolicyGenerator` or `PolicyGentemplate` CRs in the Git repository. **Additional resources** {._additional-resources} - [Removing a cluster from management](https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.9/html/clusters/cluster_mce_overview#remove-managed-cluster) - [Deprovisioning clusters](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.15/html/multicluster_engine_operator_with_red_hat_advanced_cluster_management/ibio-intro#deprovision-clusters) ## Removing obsolete content from the GitOps ZTP pipeline {#ztp-removing-obsolete-content_ztp-deploying-far-edge-sites} If a change to the `PolicyGenerator` or `PolicyGentemplate` configuration results in obsolete policies, for example, if you rename policies, use the following procedure to remove the obsolete policies. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. **Procedure** 1. Remove the affected `PolicyGenerator` or `PolicyGentemplate` files from the Git repository, commit and push to the remote repository. 2. Wait for the changes to synchronize through the application and the affected policies to be removed from the hub cluster. 3. Add the updated `PolicyGenerator` or `PolicyGentemplate` files back to the Git repository, and then commit and push to the remote repository. > [!NOTE] > Removing GitOps Zero Touch Provisioning (ZTP) policies from the Git repository, and as a result also removing them from the hub cluster, does not affect the configuration of the managed cluster. The policy and CRs managed by that policy remains in place on the managed cluster. 4. Optional: As an alternative, after making changes to `PolicyGenerator` or `PolicyGentemplate` CRs that result in obsolete policies, you can remove these policies from the hub cluster manually. You can delete policies from the RHACM console using the **Governance** tab or by running the following command: ```terminal $ oc delete policy -n ``` ## Tearing down the GitOps ZTP pipeline {#ztp-tearing-down-the-pipeline_ztp-deploying-far-edge-sites} You can remove the ArgoCD pipeline and all generated GitOps Zero Touch Provisioning (ZTP) artifacts. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. **Procedure** 1. Detach all clusters from Red Hat Advanced Cluster Management (RHACM) on the hub cluster. 2. Delete the `kustomization.yaml` file in the `deployment` directory using the following command: ```terminal $ oc delete -k out/argocd/deployment ``` 3. Commit and push your changes to the site repository.