--- title: Updating GitOps ZTP --- # Updating GitOps ZTP {#ztp-updating-gitops} You can update the GitOps Zero Touch Provisioning (ZTP) infrastructure independently from the hub cluster, Red Hat Advanced Cluster Management (RHACM), and the managed OpenShift Container Platform clusters. > [!NOTE] > You can update the Red Hat OpenShift GitOps Operator when new versions become available. When updating the GitOps ZTP plugin, review the updated files in the reference configuration and ensure that the changes meet your requirements. > [!IMPORTANT] > Using `PolicyGenTemplate` CRs to manage and deploy policies to managed clusters will be deprecated in an upcoming OpenShift Container Platform release. Equivalent and improved functionality is available using Red Hat Advanced Cluster Management (RHACM) and `PolicyGenerator` CRs. > > For more information about `PolicyGenerator` resources, see the RHACM [Integrating Policy Generator](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/governance/index#integrate-policy-generator) documentation. **Additional resources** {._additional-resources} - [Configuring managed cluster policies by using PolicyGenerator resources](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator#ztp-configuring-managed-clusters-policygenerator) - [Comparing RHACM PolicyGenerator and PolicyGenTemplate resource patching](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator#ztp-comparing-pgt-and-rhacm-pg-patching-strategies_ztp-configuring-managed-clusters-policygenerator) ## Overview of the GitOps ZTP update process {#ztp-updating-gitops-ztp_ztp-updating-gitops} You can update GitOps Zero Touch Provisioning (ZTP) for a fully operational hub cluster running an earlier version of the GitOps ZTP infrastructure. The update process avoids impact on managed clusters. > [!NOTE] > Any changes to policy settings, including adding recommended content, results in updated policies that must be rolled out to the managed clusters and reconciled. At a high level, the strategy for updating the GitOps ZTP infrastructure is as follows: 1. Label all existing clusters with the `ztp-done` label. 2. Stop the ArgoCD applications. 3. Install the new GitOps ZTP tools. 4. Update required content and optional changes in the Git repository. 5. Enable pulling the ISO images for the desired OpenShift Container Platform version. 6. Update and restart the application configuration. ## Preparing for the upgrade {#ztp-preparing-for-the-gitops-ztp-upgrade_ztp-updating-gitops} Use the following procedure to prepare your site for the GitOps Zero Touch Provisioning (ZTP) upgrade. **Procedure** 1. Get the latest version of the GitOps ZTP container that has the custom resources (CRs) used to configure Red Hat OpenShift GitOps for use with GitOps ZTP. 2. Extract the `argocd/deployment` directory by using the following commands: ```terminal $ mkdir -p ./update ``` ```terminal $ podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.22 extract /home/ztp --tar | tar x -C ./update ``` The `/update` directory contains the following subdirectories: - `update/extra-manifest`: contains the source CR files that you package into a `ConfigMap` and reference in the `ClusterInstance` CR using the `extraManifestsRefs` field. - `update/source-crs`: contains the source CR files that the `PolicyGenerator` or `PolicyGentemplate` CR uses to generate the Red Hat Advanced Cluster Management (RHACM) policies. - `update/argocd/deployment`: contains patches and YAML files to apply on the hub cluster for use in the next step of this procedure. - `update/argocd/example`: contains example `ClusterInstance` and `PolicyGenerator` or `PolicyGentemplate` files that represent the recommended configuration. 3. Update the `clusters-app.yaml` and `policies-app.yaml` files to reflect the name of your applications and the URL, branch, and path for your Git repository. If the upgrade includes changes that results in obsolete policies, the obsolete policies should be removed prior to performing the upgrade. 4. Diff the changes between the configuration and deployment source CRs in the `/update` folder and Git repo where you manage your fleet site CRs. Apply and push the required changes to your site repository. > [!IMPORTANT] > When you update GitOps ZTP to the latest version, you must apply the changes from the `update/argocd/deployment` directory to your site repository. Do not use older versions of the `argocd/deployment/` files. ## Labeling the existing clusters {#ztp-labeling-the-existing-clusters_ztp-updating-gitops} To ensure that existing clusters remain untouched by the tool updates, label all existing managed clusters with the `ztp-done` label. > [!NOTE] > This procedure only applies when updating clusters that were not provisioned with Topology Aware Lifecycle Manager (TALM). Clusters that you provision with TALM are automatically labeled with `ztp-done`. **Procedure** 1. Find a label selector that lists the managed clusters that were deployed with GitOps Zero Touch Provisioning (ZTP), such as `local-cluster!=true`: ```terminal $ oc get managedcluster -l 'local-cluster!=true' ``` 2. Ensure that the resulting list contains all the managed clusters that were deployed with GitOps ZTP, and then use that selector to add the `ztp-done` label: ```terminal $ oc label managedcluster -l 'local-cluster!=true' ztp-done= ``` ## Stopping the existing GitOps ZTP applications {#ztp-stopping-the-existing-gitops-ztp-applications_ztp-updating-gitops} Removing the existing applications ensures that any changes to existing content in the Git repository are not rolled out until the new version of the tools is available. Use the application files from the `deployment` directory. If you used custom names for the applications, update the names in these files first. **Procedure** 1. Perform a non-cascaded delete on the `clusters` application to leave all generated resources in place: ```terminal $ oc delete -f update/argocd/deployment/clusters-app.yaml ``` 2. Perform a cascaded delete on the `policies` application to remove all previous policies: ```terminal $ oc patch -f policies-app.yaml -p '{"metadata": {"finalizers": ["resources-finalizer.argocd.argoproj.io"]}}' --type merge ``` ```terminal $ oc delete -f update/argocd/deployment/policies-app.yaml ``` ## Required changes to the Git repository {#ztp-required-changes-to-the-git-repository_ztp-updating-gitops} When upgrading the `ztp-site-generate` container from an earlier release of GitOps Zero Touch Provisioning (ZTP) to 4.10 or later, there are additional requirements for the contents of the Git repository. Existing content in the repository must be updated to reflect these changes. > [!NOTE] > The following procedure assumes you are using `PolicyGenerator` resources instead of `PolicyGentemplate` resources for cluster policies management. - Make required changes to `PolicyGenerator` files: All `PolicyGenerator` files must be created in a `Namespace` prefixed with `ztp`. This ensures that the GitOps ZTP application is able to manage the policy CRs generated by GitOps ZTP without conflicting with the way Red Hat Advanced Cluster Management (RHACM) manages the policies internally. - Add the `kustomization.yaml` file to the repository: All `ClusterInstance` and `PolicyGenerator` CRs must be included in a `kustomization.yaml` file under their respective directory trees. For example: ```terminal ├── acmpolicygenerator │ ├── site1-ns.yaml │ ├── site1.yaml │ ├── site2-ns.yaml │ ├── site2.yaml │ ├── common-ns.yaml │ ├── common-ranGen.yaml │ ├── group-du-sno-ranGen-ns.yaml │ ├── group-du-sno-ranGen.yaml │ └── kustomization.yaml └── clusterinstance ├── site1.yaml ├── site2.yaml └── kustomization.yaml ``` > [!NOTE] > The files listed in the `generator` sections must contain either `ClusterInstance` or `{policy_gen_cr}` CRs only. If your existing YAML files contain other CRs, for example, `Namespace`, these other CRs must be pulled out into separate files and listed in the `resources` section. The `PolicyGenerator` kustomization file must contain all `PolicyGenerator` YAML files in the `generator` section and `Namespace` CRs in the `resources` section. For example: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization generators: - acm-common-ranGen.yaml - acm-group-du-sno-ranGen.yaml - site1.yaml - site2.yaml resources: - common-ns.yaml - acm-group-du-sno-ranGen-ns.yaml - site1-ns.yaml - site2-ns.yaml ``` The `ClusterInstance` kustomization file must contain all `ClusterInstance` YAML files in the `generator` section and any other CRs in the resources: ```terminal apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization generators: - site1.yaml - site2.yaml ``` - Remove the `pre-sync.yaml` and `post-sync.yaml` files. In OpenShift Container Platform 4.10 and later, the `pre-sync.yaml` and `post-sync.yaml` files are no longer required. The `update/deployment/kustomization.yaml` CR manages the policies deployment on the hub cluster. > [!NOTE] > There is a set of `pre-sync.yaml` and `post-sync.yaml` files under both the `ClusterInstance` and `{policy_gen_cr}` trees. - Review and incorporate recommended changes Each release may include additional recommended changes to the configuration applied to deployed clusters. Typically these changes result in lower CPU use by the OpenShift platform, additional features, or improved tuning of the platform. Review the reference `ClusterInstance` and `PolicyGenerator` CRs applicable to the types of cluster in your network. These examples can be found in the `argocd/example` directory extracted from the GitOps ZTP container. ## Installing the new GitOps ZTP applications {#ztp-installing-the-new-gitops-ztp-applications_ztp-updating-gitops} Using the extracted `argocd/deployment` directory, and after ensuring that the applications point to your site Git repository, apply the full contents of the deployment directory. Applying the full contents of the directory ensures that all necessary resources for the applications are correctly configured. **Procedure** 1. To install the GitOps ZTP plugin, patch the ArgoCD instance in the hub cluster with the relevant multicluster engine (MCE) subscription image. Customize the patch file that you previously extracted into the `out/argocd/deployment/` directory for your environment. 1. Select the `multicluster-operators-subscription` image that matches your RHACM version. - For RHACM 2.8 and 2.9, use the `registry.redhat.io/rhacm2/multicluster-operators-subscription-rhel8:v` image. - For RHACM 2.10 and later, use the `registry.redhat.io/rhacm2/multicluster-operators-subscription-rhel9:v` image. > [!IMPORTANT] > The version of the `multicluster-operators-subscription` image must match the RHACM version. Beginning with the MCE 2.10 release, RHEL 9 is the base image for `multicluster-operators-subscription` images. > > Click `[Expand for Operator list]` in the "Platform Aligned Operators" table in [OpenShift Operator Life Cycles](https://access.redhat.com/support/policy/updates/openshift_operators) to view the complete supported Operators matrix for OpenShift Container Platform. 2. Modify the `out/argocd/deployment/argocd-openshift-gitops-patch.json` file with the `multicluster-operators-subscription` image that matches your RHACM version: ```json { "args": [ "-c", "mkdir -p /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator && cp /policy-generator/PolicyGenerator-not-fips-compliant /.config/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator" ], "command": [ "/bin/bash" ], "image": "registry.redhat.io/rhacm2/multicluster-operators-subscription-rhel9:v2.10", "name": "policy-generator-install", "imagePullPolicy": "Always", "volumeMounts": [ { "mountPath": "/.config", "name": "kustomize" } ] } ``` - Optional: For RHEL 9 images, in the `args` field, change the executable path from `/policy-generator/PolicyGenerator-not-fips-compliant` to match the required universal executable for your ArgoCD version. - Match the `multicluster-operators-subscription` image to your RHACM version. In disconnected environments, replace the URL with the disconnected registry equivalent for your environment. 3. Patch the ArgoCD instance. Run the following command: ```terminal $ oc patch argocd openshift-gitops \ -n openshift-gitops --type=merge \ --patch-file out/argocd/deployment/argocd-openshift-gitops-patch.json ``` 2. In RHACM 2.7 and later, the multicluster engine enables the `cluster-proxy-addon` feature by default. Apply the following patch to disable the `cluster-proxy-addon` feature and remove the relevant hub cluster and managed pods that are responsible for this add-on. Run the following command: ```terminal $ oc patch multiclusterengines.multicluster.openshift.io multiclusterengine --type=merge --patch-file out/argocd/deployment/disable-cluster-proxy-addon.json ``` 3. Apply the pipeline configuration to your hub cluster by running the following command: ```terminal $ oc apply -k out/argocd/deployment ``` ## Pulling ISO images for the desired OpenShift Container Platform version {#ztp-pulling-ocp-images_ztp-updating-gitops} To pull ISO images for the desired OpenShift Container Platform version, update the `AgentServiceConfig` custom resource (CR) with references to the desired ISO and RootFS images that are hosted on the mirror registry HTTP server. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the hub cluster as a user with `cluster-admin` privileges. - You have RHACM with `MultiClusterHub` enabled. - You have enabled the assisted service. **Procedure** 1. Open the `AgentServiceConfig` CR to update the `spec.osImages` field by running the following command: ```terminal $ oc edit AgentServiceConfig ``` 2. Update the `spec.osImages` field in the `AgentServiceConfig` CR: ```yaml apiVersion: agent-install.openshift.io/v1beta1 kind: AgentServiceConfig metadata: name: agent spec: # ... osImages: - cpuArchitecture: x86_64 openshiftVersion: "4.22" rootFSUrl: https:////rhcos-live-rootfs.x86_64.img url: https:////rhcos-live.x86_64.iso ``` where: `` : Specifies the fully qualified domain name (FQDN) for the target mirror registry HTTP server. `` : Specifies the path to the image on the target mirror registry. 3. Save and quit the editor to apply the changes. **Additional resources** {._additional-resources} - [Enabling the assisted service](/openshift-docs-markdown/edge_computing/ztp-preparing-the-hub-cluster#enabling-assisted-installer-service-on-bare-metal_ztp-preparing-the-hub-cluster) ## Rolling out the GitOps ZTP configuration changes {#ztp-roll-out-the-configuration-changes_ztp-updating-gitops} If any configuration changes were included in the upgrade due to implementing recommended changes, the upgrade process results in a set of policy CRs on the hub cluster in the `Non-Compliant` state. With the GitOps Zero Touch Provisioning (ZTP) version 4.10 and later `ztp-site-generate` container, these policies are set to `inform` mode and are not pushed to the managed clusters without an additional step by the user. This ensures that potentially disruptive changes to the clusters can be managed in terms of when the changes are made, for example, during a maintenance window, and how many clusters are updated concurrently. To roll out the changes, create one or more `ClusterGroupUpgrade` CRs as detailed in the TALM documentation. The CR must contain the list of `Non-Compliant` policies that you want to push out to the managed clusters as well as a list or selector of which clusters should be included in the update. **Additional resources** {._additional-resources} - [About the Topology Aware Lifecycle Manager configuration](/openshift-docs-markdown/edge_computing/cnf-talm-for-cluster-upgrades#cnf-about-topology-aware-lifecycle-manager-config_cnf-topology-aware-lifecycle-manager) - [About the auto-created ClusterGroupUpgrade CR for GitOps ZTP](/openshift-docs-markdown/edge_computing/policygentemplate_for_ztp/ztp-talm-updating-managed-policies#talo-precache-autocreated-cgu-for-ztp_ztp-talm)