--- title: Updating managed clusters in a disconnected environment with PolicyGenTemplate resources and TALM --- # Updating managed clusters in a disconnected environment with PolicyGenTemplate resources and TALM {#ztp-topology-aware-lifecycle-manager} You can use the Topology Aware Lifecycle Manager (TALM) to manage the software lifecycle of managed clusters that you have deployed by using GitOps Zero Touch Provisioning (ZTP) and Topology Aware Lifecycle Manager (TALM). TALM uses Red Hat Advanced Cluster Management (RHACM) PolicyGenTemplate policies to manage and control changes applied to target clusters. > [!IMPORTANT] > Using `PolicyGenTemplate` CRs to manage and deploy policies to managed clusters will be deprecated in an upcoming OpenShift Container Platform release. Equivalent and improved functionality is available using Red Hat Advanced Cluster Management (RHACM) and `PolicyGenerator` CRs. > > For more information about `PolicyGenerator` resources, see the RHACM [Integrating Policy Generator](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/governance/index#integrate-policy-generator) documentation. **Additional resources** {._additional-resources} - [Configuring managed cluster policies by using PolicyGenerator resources](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator#ztp-configuring-managed-clusters-policygenerator) - [Comparing RHACM PolicyGenerator and PolicyGenTemplate resource patching](/openshift-docs-markdown/edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator#ztp-comparing-pgt-and-rhacm-pg-patching-strategies_ztp-configuring-managed-clusters-policygenerator) - [About the Topology Aware Lifecycle Manager](/openshift-docs-markdown/edge_computing/cnf-talm-for-cluster-upgrades#cnf-about-topology-aware-lifecycle-manager-config_cnf-topology-aware-lifecycle-manager) ## Setting up the disconnected environment {#talo-platform-prepare-for-update-env-setup_ztp-talm} TALM can perform both platform and Operator updates. You must mirror both the platform image and Operator images that you want to update to in your mirror registry before you can use TALM to update your disconnected clusters. **Procedure** - For platform updates, you must perform the following steps: 1. Mirror the required OpenShift Container Platform image repository. Ensure that the required platform image is mirrored by following the "Mirroring the OpenShift Container Platform image repository" procedure linked in the Additional resources. Save the contents of the `imageContentSources` section in the `imageContentSources.yaml` file: The following is example output: ```yaml imageContentSources: - mirrors: - mirror-ocp-registry.ibmcloud.io.cpak:5000/openshift-release-dev/openshift4 source: quay.io/openshift-release-dev/ocp-release - mirrors: - mirror-ocp-registry.ibmcloud.io.cpak:5000/openshift-release-dev/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev ``` 2. Save the image signature of the required platform image that was mirrored. You must add the image signature to the `PolicyGenTemplate` CR for platform updates. To get the image signature, perform the following steps: 1. Specify the required OpenShift Container Platform tag by running the following command: ```terminal $ OCP_RELEASE_NUMBER= ``` 2. Specify the architecture of the cluster by running the following command: ```terminal $ ARCHITECTURE= ``` - `` specifies the architecture of the cluster, such as `x86_64`, `aarch64`, `s390x`, or `ppc64le`. 3. Get the release image digest from Quay by running the following command ```terminal $ DIGEST="$(oc adm release info quay.io/openshift-release-dev/ocp-release:${OCP_RELEASE_NUMBER}-${ARCHITECTURE} | sed -n 's/Pull From: .*@//p')" ``` 4. Set the digest algorithm by running the following command: ```terminal $ DIGEST_ALGO="${DIGEST%%:*}" ``` 5. Set the digest signature by running the following command: ```terminal $ DIGEST_ENCODED="${DIGEST#*:}" ``` 6. Get the image signature from the [mirror.openshift.com](https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/) website by running the following command: ```terminal $ SIGNATURE_BASE64=$(curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIGEST_ALGO}=${DIGEST_ENCODED}/signature-1" | base64 -w0 && echo) ``` 7. Save the image signature to the `checksum-.yaml` file by running the following commands: ```terminal $ cat >checksum-${OCP_RELEASE_NUMBER}.yaml < [!NOTE] > If you want to use the image precaching for Operators and there are Operators from a different catalog source other than `redhat-operators-disconnected`, you must perform the following tasks: > > - Prepare a separate catalog source policy with the new index image or registry poll interval update for the different catalog source. > - Prepare a separate subscription policy for the required Operators that are from the different catalog source. For example, the required SRIOV-FEC Operator is available in the `certified-operators` catalog source. To update the catalog source and the Operator subscription, add the following contents to generate two policies, `du-upgrade-fec-catsrc-policy` and `du-upgrade-subscriptions-fec-policy`: ```yaml apiVersion: ran.openshift.io/v1 kind: PolicyGenTemplate metadata: name: "du-upgrade" namespace: "ztp-group-du-sno" spec: bindingRules: group-du-sno: "" mcp: "master" remediationAction: inform sourceFiles: # ... - fileName: DefaultCatsrc.yaml remediationAction: inform policyName: "fec-catsrc-policy" metadata: name: certified-operators spec: displayName: Intel SRIOV-FEC Operator image: registry.example.com:5000/olm/far-edge-sriov-fec:v4.10 updateStrategy: registryPoll: interval: 10m - fileName: AcceleratorsSubscription.yaml policyName: "subscriptions-fec-policy" spec: channel: "stable" source: certified-operators ``` 3. Remove the specified subscriptions channels in the common `PolicyGenTemplate` CR, if they exist. The default subscriptions channels from the GitOps ZTP image are used for the update. > [!NOTE] > The default channel for the Operators applied through GitOps ZTP 4.22 is `stable`, except for the `performance-addon-operator`. As of OpenShift Container Platform 4.11, the `performance-addon-operator` functionality was moved to the `node-tuning-operator`. For the 4.10 release, the default channel for PAO is `v4.10`. You can also specify the default channels in the common `PolicyGenTemplate` CR. 4. Push the `PolicyGenTemplate` CRs updates to the GitOps ZTP Git repository. ArgoCD pulls the changes from the Git repository and generates the policies on the hub cluster. 5. Check the created policies by running the following command: ```terminal $ oc get policies -A | grep -E "catsrc-policy|subscription" ``` 2. Apply the required catalog source updates before starting the Operator update. 1. Save the content of the `ClusterGroupUpgrade` CR named `operator-upgrade-prep` with the catalog source policies and the target managed clusters to the `cgu-operator-upgrade-prep.yml` file: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: name: cgu-operator-upgrade-prep namespace: default spec: clusters: - spoke1 enable: true managedPolicies: - du-upgrade-operator-catsrc-policy remediationStrategy: maxConcurrency: 1 ``` 2. Apply the policy to the hub cluster by running the following command: ```terminal $ oc apply -f cgu-operator-upgrade-prep.yml ``` 3. Monitor the update process. Upon completion, ensure that the policy is compliant by running the following command: ```terminal $ oc get policies -A | grep -E "catsrc-policy" ``` 3. Create the `ClusterGroupUpgrade` CR for the Operator update with the `spec.enable` field set to `false`. 1. Save the content of the Operator update `ClusterGroupUpgrade` CR with the `du-upgrade-operator-catsrc-policy` policy and the subscription policies created from the common `PolicyGenTemplate` and the target clusters to the `cgu-operator-upgrade.yml` file, as shown in the following example: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: name: cgu-operator-upgrade namespace: default spec: managedPolicies: - du-upgrade-operator-catsrc-policy - common-subscriptions-policy preCaching: false clusters: - spoke1 remediationStrategy: maxConcurrency: 1 enable: false ``` - `du-upgrade-operator-catsrc-policy` is needed by the image precaching feature to retrieve the Operator images from the catalog source. - `common-subscriptions-policy` contains Operator subscriptions. If you have followed the structure and content of the reference `PolicyGenTemplates`, all Operator subscriptions are grouped into the `common-subscriptions-policy` policy. > [!NOTE] > One `ClusterGroupUpgrade` CR can only precache the images of the required Operators defined in the subscription policy from one catalog source included in the `ClusterGroupUpgrade` CR. If the required Operators are from different catalog sources, such as in the example of the SRIOV-FEC Operator, another `ClusterGroupUpgrade` CR must be created with `du-upgrade-fec-catsrc-policy` and `du-upgrade-subscriptions-fec-policy` policies for the SRIOV-FEC Operator images precaching and update. 2. Apply the `ClusterGroupUpgrade` CR to the hub cluster by running the following command: ```terminal $ oc apply -f cgu-operator-upgrade.yml ``` 4. Optional: Precache the images for the Operator update. 1. Before starting image precaching, verify the subscription policy is `NonCompliant` at this point by running the following command: ```terminal $ oc get policy common-subscriptions-policy -n ``` The following is example output: ```terminal NAME REMEDIATION ACTION COMPLIANCE STATE AGE common-subscriptions-policy inform NonCompliant 27d ``` 2. Enable precaching in the `ClusterGroupUpgrade` CR by running the following command: ```terminal $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-operator-upgrade \ --patch '{"spec":{"preCaching": true}}' --type=merge ``` 3. Monitor the process and wait for the precaching to complete. Check the status of precaching by running the following command on the managed cluster: ```terminal $ oc get cgu cgu-operator-upgrade -o jsonpath='{.status.precaching.status}' ``` 4. Check if the precaching is completed before starting the update by running the following command: ```terminal $ oc get cgu -n default cgu-operator-upgrade -ojsonpath='{.status.conditions}' | jq ``` The following is example output: ```json [ { "lastTransitionTime": "2022-03-08T20:49:08.000Z", "message": "The ClusterGroupUpgrade CR is not enabled", "reason": "UpgradeNotStarted", "status": "False", "type": "Ready" }, { "lastTransitionTime": "2022-03-08T20:55:30.000Z", "message": "Precaching is completed", "reason": "PrecachingCompleted", "status": "True", "type": "PrecachingDone" } ] ``` 5. Start the Operator update. 1. Enable the `cgu-operator-upgrade` `ClusterGroupUpgrade` CR and disable precaching to start the Operator update by running the following command: ```terminal $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-operator-upgrade \ --patch '{"spec":{"enable":true, "preCaching": false}}' --type=merge ``` 2. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command: ```terminal $ oc get policies --all-namespaces ``` **Additional resources** {._additional-resources} - [Upgrading GitOps ZTP](/openshift-docs-markdown/edge_computing/ztp-updating-gitops#ztp-updating-gitops) ## Troubleshooting missed Operator updates with PolicyGenTemplate CRs {#cnf-topology-aware-lifecycle-manager-operator-troubleshooting-PolicyGenTemplate_ztp-talm} In some scenarios, Topology Aware Lifecycle Manager (TALM) might miss Operator updates due to an out-of-date policy compliance state. After a catalog source update, it takes time for the Operator Lifecycle Manager (OLM) to update the subscription status. The status of the subscription policy might continue to show as compliant while TALM decides whether remediation is needed. As a result, the Operator specified in the subscription policy does not get upgraded. To avoid this scenario, add another catalog source configuration to the `PolicyGenTemplate` and specify this configuration in the subscription for any Operators that require an update. **Procedure** 1. Add a catalog source configuration in the `PolicyGenTemplate` resource: ```yaml - fileName: DefaultCatsrc.yaml remediationAction: inform policyName: "operator-catsrc-policy" metadata: name: redhat-operators-disconnected spec: displayName: Red Hat Operators Catalog image: registry.example.com:5000/olm/redhat-operators-disconnected:v{product-version} updateStrategy: registryPoll: interval: 1h status: connectionState: lastObservedState: READY - fileName: DefaultCatsrc.yaml remediationAction: inform policyName: "operator-catsrc-policy" metadata: name: redhat-operators-disconnected-v2 spec: displayName: Red Hat Operators Catalog v2 image: registry.example.com:5000/olm/redhat-operators-disconnected: updateStrategy: registryPoll: interval: 1h status: connectionState: lastObservedState: READY ``` - `name` - Update the name for the new configuration. - `displayName` - Update the display name for the new configuration. - `image` - Update the index image URL. This `fileName.spec.image` field overrides any configuration in the `DefaultCatsrc.yaml` file. 2. Update the `Subscription` resource to point to the new configuration for Operators that require an update: ```yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: operator-subscription namespace: operator-namspace # ... spec: source: redhat-operators-disconnected-v2 # ... ``` - `redhat-operators-disconnected-v2` specifies the name of the additional catalog source configuration that you defined in the `PolicyGenTemplate` resource. ## Performing a platform and an Operator update together {#talo-operator-and-platform-update_ztp-talm} You can perform a platform and an Operator update at the same time. **Prerequisites** - Install the Topology Aware Lifecycle Manager (TALM). - Update GitOps Zero Touch Provisioning (ZTP) to the latest version. - Provision one or more managed clusters with GitOps ZTP. - Log in as a user with `cluster-admin` privileges. - Create RHACM policies in the hub cluster. **Procedure** 1. Create the `PolicyGenTemplate` CR for the updates by following the steps described in the "Performing a platform update" and "Performing an Operator update" sections. 2. Apply the prep work for the platform and the Operator update. 1. Save the content of the `ClusterGroupUpgrade` CR with the policies for platform update preparation work, catalog source updates, and target clusters to the `cgu-platform-operator-upgrade-prep.yml` file, for example: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: name: cgu-platform-operator-upgrade-prep namespace: default spec: managedPolicies: - du-upgrade-platform-upgrade-prep - du-upgrade-operator-catsrc-policy clusterSelector: - group-du-sno remediationStrategy: maxConcurrency: 10 enable: true ``` 2. Apply the `cgu-platform-operator-upgrade-prep.yml` file to the hub cluster by running the following command: ```terminal $ oc apply -f cgu-platform-operator-upgrade-prep.yml ``` 3. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command: ```terminal $ oc get policies --all-namespaces ``` 3. Create the `ClusterGroupUpdate` CR for the platform and the Operator update with the `spec.enable` field set to `false`. 1. Save the contents of the platform and Operator update `ClusterGroupUpdate` CR with the policies and the target clusters to the `cgu-platform-operator-upgrade.yml` file, as shown in the following example: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: name: cgu-du-upgrade namespace: default spec: managedPolicies: - du-upgrade-platform-upgrade - du-upgrade-operator-catsrc-policy - common-subscriptions-policy preCaching: true clusterSelector: - group-du-sno remediationStrategy: maxConcurrency: 1 enable: false ``` - `du-upgrade-platform-upgrade` is the platform update policy. - `du-upgrade-operator-catsrc-policy` is the policy containing the catalog source information for the Operators to be updated. It is needed for the precaching feature to determine which Operator images to download to the managed cluster. - `common-subscriptions-policy` is the policy to update the Operators. 2. Apply the `cgu-platform-operator-upgrade.yml` file to the hub cluster by running the following command: ```terminal $ oc apply -f cgu-platform-operator-upgrade.yml ``` 4. Optional: Precache the images for the platform and the Operator update. 1. Enable precaching in the `ClusterGroupUpgrade` CR by running the following command: ```terminal $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-du-upgrade \ --patch '{"spec":{"preCaching": true}}' --type=merge ``` 2. Monitor the update process and wait for the precaching to complete. Check the status of precaching by running the following command on the managed cluster: ```terminal $ oc get jobs,pods -n openshift-talm-pre-cache ``` 3. Check if the precaching is completed before starting the update by running the following command: ```terminal $ oc get cgu cgu-du-upgrade -ojsonpath='{.status.conditions}' ``` 5. Start the platform and Operator update. 1. Enable the `cgu-du-upgrade` `ClusterGroupUpgrade` CR to start the platform and the Operator update by running the following command: ```terminal $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-du-upgrade \ --patch '{"spec":{"enable":true, "preCaching": false}}' --type=merge ``` 2. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command: ```terminal $ oc get policies --all-namespaces ``` > [!NOTE] > The CRs for the platform and Operator updates can be created from the beginning by configuring the setting to `spec.enable: true`. In this case, the update starts immediately after precaching completes and there is no need to manually enable the CR. > > Both precaching and the update create extra resources, such as policies, placement bindings, placement rules, managed cluster actions, and managed cluster view, to help complete the procedures. Setting the `afterCompletion.deleteObjects` field to `true` deletes all these resources after the updates complete. ## Removing Performance Addon Operator subscriptions from deployed clusters with PolicyGenTemplate CRs {#talm-pao-update-PolicyGenTemplate_ztp-talm} In earlier versions of OpenShift Container Platform, the Performance Addon Operator provided automatic, low latency performance tuning for applications. In OpenShift Container Platform 4.11 or later, these functions are part of the Node Tuning Operator. Do not install the Performance Addon Operator on clusters running OpenShift Container Platform 4.11 or later. If you upgrade to OpenShift Container Platform 4.11 or later, the Node Tuning Operator automatically removes the Performance Addon Operator. > [!NOTE] > You need to remove any policies that create Performance Addon Operator subscriptions to prevent a re-installation of the Operator. The reference DU profile includes the Performance Addon Operator in the `PolicyGenTemplate` CR `truecommon-ranGen.yaml`. To remove the subscription from deployed managed clusters, you must update `truecommon-ranGen.yaml`. > [!NOTE] > If you install Performance Addon Operator 4.10.3-5 or later on OpenShift Container Platform 4.11 or later, the Performance Addon Operator detects the cluster version and automatically hibernates to avoid interfering with the Node Tuning Operator functions. However, to ensure best performance, remove the Performance Addon Operator from your OpenShift Container Platform 4.11 clusters. **Prerequisites** - Create a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and be defined as a source repository for ArgoCD. - Update to OpenShift Container Platform 4.11 or later. - Log in as a user with `cluster-admin` privileges. **Procedure** 1. Change the `complianceType` to `mustnothave` for the Performance Addon Operator namespace, Operator group, and subscription in the `truecommon-ranGen.yaml` file. ```yaml - fileName: PaoSubscriptionNS.yaml policyName: "subscriptions-policy" complianceType: mustnothave - fileName: PaoSubscriptionOperGroup.yaml policyName: "subscriptions-policy" complianceType: mustnothave - fileName: PaoSubscription.yaml policyName: "subscriptions-policy" complianceType: mustnothave ``` 2. Merge the changes with your custom site repository and wait for the ArgoCD application to synchronize the change to the hub cluster. The status of the `common-subscriptions-policy` policy changes to `Non-Compliant`. 3. Apply the change to your target clusters by using the Topology Aware Lifecycle Manager. For more information about rolling out configuration changes, see the "Additional resources" section. 4. Monitor the process. When the status of the `common-subscriptions-policy` policy for a target cluster is `Compliant`, the Performance Addon Operator has been removed from the cluster. Get the status of the `common-subscriptions-policy` by running the following command: ```terminal $ oc get policy -n ztp-common common-subscriptions-policy ``` 5. Delete the Performance Addon Operator namespace, Operator group and subscription CRs from `spec.sourceFiles` in the `truecommon-ranGen.yaml` file. 6. Merge the changes with your custom site repository and wait for the ArgoCD application to synchronize the change to the hub cluster. The policy remains compliant. ## Precaching user-specified images with TALM on single-node OpenShift clusters {#talm-prechache-user-specified-images-concept_ztp-talm} You can precache application-specific workload images on single-node OpenShift clusters before updating your applications. You can specify the configuration options for the precaching jobs by using the following custom resources (CR): - `PreCachingConfig` CR - `ClusterGroupUpgrade` CR TALM derives the platform image from the `ClusterVersion` object in the managed policies. TALM derives Operator index images from `CatalogSource` objects that the managed policies reference. > [!NOTE] > All fields in the `PreCachingConfig` CR are optional. The following example shows a `PreCachingConfig` CR: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: PreCachingConfig metadata: name: exampleconfig namespace: exampleconfig-ns spec: overrides: operatorsPackagesAndChannels: - local-storage-operator: stable - ptp-operator: stable - sriov-network-operator: stable spaceRequired: 30 Gi excludePrecachePatterns: - aws - vsphere additionalImages: - quay.io/exampleconfig/application1@sha256:3d5800990dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47e2e1ef - quay.io/exampleconfig/application2@sha256:3d5800123dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47adfaef - quay.io/exampleconfig/applicationN@sha256:4fe1334adfafadsf987123adfffdaf1243340adfafdedga0991234afdadfsa09 ``` - `overrides` - Specifies Operator packages and channels to precache instead of the values that TALM derives from the managed policies. The only supported override is `operatorsPackagesAndChannels`. TALM ignores the deprecated `platformImage` and `operatorsIndexes` override fields if they are present. - `spaceRequired` - Specifies the minimum required disk space on the cluster. If unspecified, TALM defines a default value for OpenShift Container Platform images. The disk space field must include an integer value and the storage unit. For example: `40 GiB`, `200 MB`, `1 TiB`. - `excludePrecachePatterns` - Specifies the images to exclude from precaching based on image name matching. - `additionalImages` - Specifies the list of additional images to precache. The following example shows a `ClusterGroupUpgrade` CR with a `PreCachingConfig` CR reference: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: name: cgu spec: preCaching: true preCachingConfigRef: name: exampleconfig namespace: exampleconfig-ns ``` - `preCaching` set to `true` enables the precaching job. - `preCachingConfigRef.name` specifies the `PreCachingConfig` CR that you want to use. - `preCachingConfigRef.namespace` specifies the namespace of the `PreCachingConfig` CR that you want to use. ### Creating the custom resources for precaching {#talm-prechache-user-specified-images-preparing-crs_ztp-talm} You must create the `PreCachingConfig` CR before or concurrently with the `ClusterGroupUpgrade` CR. **Procedure** 1. Create the `PreCachingConfig` CR with the list of additional images you want to precache. ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: PreCachingConfig metadata: name: exampleconfig namespace: default spec: # ... spaceRequired: 30Gi additionalImages: - quay.io/exampleconfig/application1@sha256:3d5800990dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47e2e1ef - quay.io/exampleconfig/application2@sha256:3d5800123dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47adfaef - quay.io/exampleconfig/applicationN@sha256:4fe1334adfafadsf987123adfffdaf1243340adfafdedga0991234afdadfsa09 ``` - `namespace` must be accessible to the hub cluster. - `spaceRequired` - It is recommended to set the minimum disk space required field to ensure that there is sufficient storage space for the precached images. 2. Create a `ClusterGroupUpgrade` CR with the `preCaching` field set to `true` and specify the `PreCachingConfig` CR created in the previous step: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: name: cgu namespace: default spec: clusters: - sno1 - sno2 preCaching: true preCachingConfigRef: - name: exampleconfig namespace: default managedPolicies: - du-upgrade-platform-upgrade - du-upgrade-operator-catsrc-policy - common-subscriptions-policy remediationStrategy: timeout: 240 ``` > [!WARNING] > Once you install the images on the cluster, you cannot change or delete them. 3. When you want to start precaching the images, apply the `ClusterGroupUpgrade` CR by running the following command: ```terminal $ oc apply -f cgu.yaml ``` TALM verifies the `ClusterGroupUpgrade` CR. From this point, you can continue with the TALM precaching workflow. > [!NOTE] > All sites are precached concurrently. **Verification** 1. Check the precaching status on the hub cluster where the `ClusterGroupUpgrade` CR is applied by running the following command: ```terminal $ oc get cgu -n -oyaml ``` The following example shows the derived precaching specification. The `platformImage` and `operatorsIndexes` values come from the managed policies, not from `PreCachingConfig` overrides. ```yaml precaching: spec: platformImage: quay.io/openshift-release-dev/ocp-release@sha256:3d5800990dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47e2e1ef operatorsIndexes: - registry.example.com:5000/custom-redhat-operators:1.0.0 operatorsPackagesAndChannels: - local-storage-operator: stable - ptp-operator: stable - sriov-network-operator: stable excludePrecachePatterns: - aws - vsphere additionalImages: - quay.io/exampleconfig/application1@sha256:3d5800990dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47e2e1ef - quay.io/exampleconfig/application2@sha256:3d5800123dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47adfaef - quay.io/exampleconfig/applicationN@sha256:4fe1334adfafadsf987123adfffdaf1243340adfafdedga0991234afdadfsa09 spaceRequired: "30" status: sno1: Starting sno2: Starting ``` The precaching configurations are validated by checking if the managed policies exist. Valid configurations of the `ClusterGroupUpgrade` and the `PreCachingConfig` CRs result in the following statuses: The following example shows the output of valid CRs: ```yaml - lastTransitionTime: "2023-01-01T00:00:01Z" message: All selected clusters are valid reason: ClusterSelectionCompleted status: "True" type: ClusterSelected - lastTransitionTime: "2023-01-01T00:00:02Z" message: Completed validation reason: ValidationCompleted status: "True" type: Validated - lastTransitionTime: "2023-01-01T00:00:03Z" message: Precaching spec is valid and consistent reason: PrecacheSpecIsWellFormed status: "True" type: PrecacheSpecValid - lastTransitionTime: "2023-01-01T00:00:04Z" message: Precaching in progress for 1 clusters reason: InProgress status: "False" type: PrecachingSucceeded ``` The following example shows an invalid `PreCachingConfig` CR: ```yaml Type: "PrecacheSpecValid" Status: False, Reason: "PrecacheSpecIncomplete" Message: "Precaching spec is incomplete: failed to get PreCachingConfig resource due to PreCachingConfig.ran.openshift.io "" not found" ``` 2. You can find the precaching job by running the following command on the managed cluster: ```terminal $ oc get jobs -n openshift-talo-pre-cache ``` The following example shows a precaching job in progress: ```terminal NAME COMPLETIONS DURATION AGE pre-cache 0/1 1s 1s ``` 3. You can check the status of the pod created for the precaching job by running the following command: ```terminal $ oc describe pod pre-cache -n openshift-talo-pre-cache ``` The following example shows a precaching job in progress: ```terminal Type Reason Age From Message Normal SuccesfulCreate 19s job-controller Created pod: pre-cache-abcd1 ``` 4. You can get live updates on the status of the job by running the following command: ```terminal $ oc logs -f pre-cache-abcd1 -n openshift-talo-pre-cache ``` 5. To verify the precache job is successfully completed, run the following command: ```terminal $ oc describe pod pre-cache -n openshift-talo-pre-cache ``` The following example shows a completed precache job: ```terminal Type Reason Age From Message Normal SuccesfulCreate 5m19s job-controller Created pod: pre-cache-abcd1 Normal Completed 19s job-controller Job completed ``` 6. To verify that the images are successfully precached on the single-node OpenShift, do the following: 1. Enter into the node in debug mode: ```terminal $ oc debug node/cnfdf00.example.lab ``` 2. Change root to `host`: ```terminal $ chroot /host/ ``` 3. Search for the required images: ```terminal $ sudo podman images | grep ``` **Additional resources** {._additional-resources} - [Using the container image precache feature](/openshift-docs-markdown/edge_computing/cnf-talm-for-cluster-upgrades#talo-precache-feature-concept_cnf-topology-aware-lifecycle-manager) ## About the auto-created ClusterGroupUpgrade CR for GitOps ZTP {#talo-precache-autocreated-cgu-for-ztp_ztp-talm} TALM has a controller called `ManagedClusterForCGU` that monitors the `Ready` state of the `ManagedCluster` CRs on the hub cluster and creates the `ClusterGroupUpgrade` CRs for GitOps Zero Touch Provisioning (ZTP). For any managed cluster in the `Ready` state without a `ztp-done` label applied, the `ManagedClusterForCGU` controller automatically creates a `ClusterGroupUpgrade` CR in the `ztp-install` namespace with its associated RHACM policies that are created during the GitOps ZTP process. TALM then remediates the set of configuration policies that are listed in the auto-created `ClusterGroupUpgrade` CR to push the configuration CRs to the managed cluster. If there are no policies for the managed cluster at the time when the cluster becomes `Ready`, a `ClusterGroupUpgrade` CR with no policies is created. Upon completion of the `ClusterGroupUpgrade` the managed cluster is labeled as `ztp-done`. If there are policies that you want to apply for that managed cluster, manually create a `ClusterGroupUpgrade` as a Day 2 operation. **Procedure** - View the auto-created `ClusterGroupUpgrade` CR for GitOps ZTP: The following example shows an auto-created `ClusterGroupUpgrade` CR for GitOps ZTP: ```yaml apiVersion: ran.openshift.io/v1alpha1 kind: ClusterGroupUpgrade metadata: generation: 1 name: spoke1 namespace: ztp-install ownerReferences: - apiVersion: cluster.open-cluster-management.io/v1 blockOwnerDeletion: true controller: true kind: ManagedCluster name: spoke1 uid: 98fdb9b2-51ee-4ee7-8f57-a84f7f35b9d5 resourceVersion: "46666836" uid: b8be9cd2-764f-4a62-87d6-6b767852c7da spec: actions: afterCompletion: addClusterLabels: ztp-done: "" deleteClusterLabels: ztp-running: "" deleteObjects: true beforeEnable: addClusterLabels: ztp-running: "" clusters: - spoke1 enable: true managedPolicies: - common-spoke1-config-policy - common-spoke1-subscriptions-policy - group-spoke1-config-policy - spoke1-config-policy - group-spoke1-validator-du-policy preCaching: false remediationStrategy: maxConcurrency: 1 timeout: 240 ``` - `ztp-done: ""` is applied to the managed cluster when TALM completes the cluster configuration. - `ztp-running: ""` is applied to the managed cluster when TALM starts deploying the configuration policies.