--- title: Expanding single-node OpenShift clusters with GitOps ZTP --- # Expanding single-node OpenShift clusters with GitOps ZTP {#ztp-sno-additional-worker-node} You can expand single-node OpenShift clusters with GitOps Zero Touch Provisioning (ZTP). When you add a worker node to single-node OpenShift clusters, the original single-node OpenShift cluster retains the control plane node role. Adding a worker node does not require any downtime for the existing single-node OpenShift cluster. > [!NOTE] > You can only expand a single-node OpenShift cluster with one additional worker node. It is not recommended to expand a single-node OpenShift cluster with more than one worker node. If you require workload partitioning on the worker node, you must deploy and remediate the managed cluster policies on the hub cluster before installing the node. This way, the workload partitioning `MachineConfig` objects are rendered and associated with the `worker` machine config pool before the GitOps ZTP workflow applies the `MachineConfig` ignition file to the worker node. It is recommended that you first remediate the policies, and then install the worker node. If you create the workload partitioning manifests after installing the worker node, you must drain the node manually and delete all the pods managed by daemon sets. When the managing daemon sets create the new pods, the new pods undergo the workload partitioning process. > [!IMPORTANT] > Adding a worker node to single-node OpenShift clusters with GitOps ZTP is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. > > For more information about the support scope of Red Hat Technology Preview features, see [Technology Preview Features Support Scope](https://access.redhat.com/support/offerings/techpreview/). **Additional resources** {._additional-resources} - [Reference configuration for deploying vDUs on single-node OpenShift](/openshift-docs-markdown/edge_computing/ztp-reference-cluster-configuration-for-vdu#sno-configure-for-vdu) - [Adding worker nodes to single-node OpenShift clusters](/openshift-docs-markdown/nodes/nodes/nodes-sno-worker-nodes#nodes-sno-worker-nodes) - [Removing managed cluster nodes by using the command line interface](https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.10/html/clusters/cluster_mce_overview#auto-remove-host-steps-cli) ## Applying profiles to the worker node with PolicyGenerator or PolicyGenTemplate resources {#ztp-additional-worker-apply-du-profile_sno-additional-worker} You can configure the additional worker node with a DU profile. You can apply a RAN distributed unit (DU) profile to the worker node cluster using the GitOps Zero Touch Provisioning (ZTP) common, group, and site-specific `PolicyGenerator` or `PolicyGenTemplate` resources. The GitOps ZTP pipeline that is linked to the ArgoCD `policies` application includes the following CRs that you can find in the relevant `out/argocd/example` folder when you extract the `ztp-site-generate` container: /acmpolicygenerator resources : - `acm-common-ranGen.yaml` - `acm-group-du-sno-ranGen.yaml` - `acm-example-sno-site.yaml` - `ns.yaml` - `kustomization.yaml` /policygentemplates resources : - `common-ranGen.yaml` - `group-du-sno-ranGen.yaml` - `example-sno-site.yaml` - `ns.yaml` - `kustomization.yaml` Configuring the DU profile on the worker node is considered an upgrade. To initiate the upgrade flow, you must update the existing policies or create additional ones. Then, you must create a `ClusterGroupUpgrade` CR to reconcile the policies in the group of clusters. ## Ensuring PTP and SR-IOV daemon selector compatibility {#ztp-additional-worker-daemon-selector-comp_sno-additional-worker} If the DU profile was deployed using the GitOps Zero Touch Provisioning (ZTP) plugin version 4.11 or earlier, the PTP and SR-IOV Operators might be configured to place the daemons only on nodes labeled as `master`. This configuration prevents the PTP and SR-IOV daemons from operating on the worker node. If the PTP and SR-IOV daemon node selectors are incorrectly configured on your system, you must change the daemons before proceeding with the worker DU profile configuration. **Procedure** 1. Check the daemon node selector settings of the PTP Operator on one of the spoke clusters: ```terminal $ oc get ptpoperatorconfig/default -n openshift-ptp -ojsonpath='{.spec}' | jq ``` The following is example output for the PTP Operator: ```json {"daemonNodeSelector":{"node-role.kubernetes.io/master":""}} ``` - If the node selector is set to `master`, the spoke was deployed with the version of the GitOps ZTP plugin that requires changes. 2. Check the daemon node selector settings of the SR-IOV Operator on one of the spoke clusters: ```terminal $ oc get sriovoperatorconfig/default -n \ openshift-sriov-network-operator -ojsonpath='{.spec}' | jq ``` The following is example output for the SR-IOV Operator: ```json {"configDaemonNodeSelector":{"node-role.kubernetes.io/worker":""},"disableDrain":false,"enableInjector":true,"enableOperatorWebhook":true} ``` - If the node selector is set to `master`, the spoke was deployed with the version of the GitOps ZTP plugin that requires changes. 3. In the group policy, add the following `complianceType` and `spec` entries: ```yaml spec: - fileName: PtpOperatorConfig.yaml policyName: "config-policy" complianceType: mustonlyhave spec: daemonNodeSelector: node-role.kubernetes.io/worker: "" - fileName: SriovOperatorConfig.yaml policyName: "config-policy" complianceType: mustonlyhave spec: configDaemonNodeSelector: node-role.kubernetes.io/worker: "" ``` > [!IMPORTANT] > Changing the `daemonNodeSelector` field causes temporary PTP synchronization loss and SR-IOV connectivity loss. 4. Commit the changes in Git, and then push to the Git repository being monitored by the GitOps ZTP ArgoCD application. ## PTP and SR-IOV node selector compatibility {#ztp-additional-worker-node-selector-comp_sno-additional-worker} The PTP configuration resources and SR-IOV network node policies use `node-role.kubernetes.io/master: ""` as the node selector. If the additional worker node has the same NIC configuration as the control plane node, the policies used to configure the control plane node can be reused for the worker node. However, the node selector must be changed to select both node types, for example with the `"node-role.kubernetes.io/worker"` label. ## Using PolicyGenerator CRs to apply worker node policies to the worker node {#ztp-additional-worker-policies-PolicyGenerator_sno-additional-worker} You can create policies for the additional worker node by using `PolicyGenerator` CRs. **Procedure** 1. Create the following `PolicyGenerator` CR: ```yaml apiVersion: policy.open-cluster-management.io/v1 kind: PolicyGenerator metadata: name: example-sno-workers placementBindingDefaults: name: example-sno-workers-placement-binding policyDefaults: namespace: example-sno placement: labelSelector: matchExpressions: - key: sites operator: In values: - example-sno remediationAction: inform severity: low namespaceSelector: exclude: - kube-* include: - '*' evaluationInterval: compliant: 10m noncompliant: 10s policies: - name: example-sno-workers-config-policy policyAnnotations: ran.openshift.io/ztp-deploy-wave: "10" manifests: - path: source-crs/PerformanceProfile-MCP-worker.yaml patches: - metadata: name: openshift-worker-node-performance-profile spec: cpu: isolated: 4-47 reserved: 0-3 hugepages: defaultHugepagesSize: 1G pages: - count: 32 size: 1G realTimeKernel: enabled: true - path: source-crs/TunedPerformancePatch-MCP-worker.yaml patches: - metadata: name: performance-patch-worker spec: profile: - data: | [main] summary=Configuration changes profile inherited from performance created tuned include=openshift-node-performance-openshift-worker-node-performance-profile [bootloader] cmdline_crash=nohz_full=4-47 [sysctl] kernel.timer_migration=1 [scheduler] group.ice-ptp=0:f:10:*:ice-ptp.* [service] service.stalld=start,enable service.chronyd=stop,disable name: performance-patch-worker recommend: - profile: performance-patch-worker ``` - `policyDefaults.placement.labelSelector.matchExpressions` — The policies are applied to all clusters with this label. - `spec.cpu.isolated` and `spec.cpu.reserved` in the `PerformanceProfile-MCP-worker.yaml` manifest — These fields must be configured for each specific hardware platform. - `cmdline_crash` in the `TunedPerformancePatch-MCP-worker.yaml` manifest — The `nohz_full` CPU set must match the `cpu.isolated` set in the `PerformanceProfile` section. You can generate the content of `crio` and `kubelet` configuration files. 2. Add the created policy template to the Git repository monitored by the ArgoCD `policies` application. 3. Add the policy in the `kustomization.yaml` file. 4. Commit the changes in Git, and then push to the Git repository being monitored by the GitOps ZTP ArgoCD application. 5. To remediate the new policies to your spoke cluster, create a TALM custom resource: ```terminal $ cat <.` **Procedure** 1. If you deployed your cluster by using the `example-sno.yaml` `ClusterInstance` CR, add your new worker node to the `spec.nodes` list: ```yaml apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "example-sno" namespace: "example-sno" spec: # ... existing cluster configuration ... nodes: - hostName: "example-sno.example.com" role: "master" # ... existing master node configuration ... - hostName: "example-node2.example.com" role: "worker" bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1" bmcCredentialsName: name: "example-node2-bmh-secret" bootMACAddress: "AA:BB:CC:DD:EE:11" bootMode: "UEFI" nodeNetwork: interfaces: - name: eno1 macAddress: "AA:BB:CC:DD:EE:11" config: interfaces: - name: eno1 type: ethernet state: up macAddress: "AA:BB:CC:DD:EE:11" ipv4: enabled: false ipv6: enabled: true address: - ip: 1111:2222:3333:4444::1 prefix-length: 64 dns-resolver: config: search: - example.com server: - 1111:2222:3333:4444::2 routes: config: - destination: ::/0 next-hop-interface: eno1 next-hop-address: 1111:2222:3333:4444::1 table-id: 254 ``` 2. Create a BMC authentication secret for the new host, as referenced by the `bmcCredentialsName` field in the `spec.nodes` section of your `ClusterInstance` CR: ```yaml apiVersion: v1 data: password: "password" username: "username" kind: Secret metadata: name: "example-node2-bmh-secret" namespace: example-sno type: Opaque ``` 3. Commit the changes in Git, and then push to the Git repository that is being monitored by the GitOps ZTP ArgoCD application. When the ArgoCD `cluster` application synchronizes, two new manifests appear on the hub cluster generated by the SiteConfig Operator: - `BareMetalHost` - `NMStateConfig` > [!IMPORTANT] > The `cpuset` field should not be configured for the worker node. Workload partitioning for the worker node is added through management policies after the node installation is complete. **Verification** You can monitor the installation process in several ways. - Check if the preprovisioning images are created by running the following command: ```terminal $ oc get ppimg -n example-sno ``` The following is example output: ```terminal NAMESPACE NAME READY REASON example-sno example-sno True ImageCreated example-sno example-node2 True ImageCreated ``` - Check the state of the bare-metal hosts: ```terminal $ oc get bmh -n example-sno ``` The following is example output: ```terminal NAME STATE CONSUMER ONLINE ERROR AGE example-sno provisioned true 69m example-node2 provisioning true 4m50s ``` - The `provisioning` state indicates that node booting from the installation media is in progress. - Continuously monitor the installation process: 1. Watch the agent install process by running the following command: ```terminal $ oc get agent -n example-sno --watch ``` The following is example output: ```terminal NAME CLUSTER APPROVED ROLE STAGE 671bc05d-5358-8940-ec12-d9ad22804faa example-sno true master Done [...] 14fd821b-a35d-9cba-7978-00ddf535ff37 example-sno true worker Starting installation 14fd821b-a35d-9cba-7978-00ddf535ff37 example-sno true worker Installing 14fd821b-a35d-9cba-7978-00ddf535ff37 example-sno true worker Writing image to disk [...] 14fd821b-a35d-9cba-7978-00ddf535ff37 example-sno true worker Waiting for control plane [...] 14fd821b-a35d-9cba-7978-00ddf535ff37 example-sno true worker Rebooting 14fd821b-a35d-9cba-7978-00ddf535ff37 example-sno true worker Done ``` 2. When the worker node installation is finished, the worker node certificates are approved automatically. At this point, the worker appears in the `ManagedClusterInfo` status. Run the following command to see the status: ```terminal $ oc get managedclusterinfo/example-sno -n example-sno -o \ jsonpath='{range .status.nodeList[*]}{.name}{"\t"}{.conditions}{"\t"}{.labels}{"\n"}{end}' ``` The following is example output: ```terminal example-sno [{"status":"True","type":"Ready"}] {"node-role.kubernetes.io/master":"","node-role.kubernetes.io/worker":""} example-node2 [{"status":"True","type":"Ready"}] {"node-role.kubernetes.io/worker":""} ```