OADP monitoring¶

Monitor OADP operations by using the OpenShift Container Platform monitoring stack to create service monitors, configure alerting rules, and view metrics. This helps you track backup and restore performance, manage clusters, and receive alerts for important events.

OADP monitoring setup¶

Set up OADP monitoring by enabling User Workload Monitoring and configuring the OpenShift Container Platform monitoring stack to retrieve Velero metrics. This helps you create alerting rules, query metrics, and optionally visualize data by using Prometheus-compatible tools such as Grafana.

Monitoring metrics requires enabling monitoring for the user-defined projects and creating a ServiceMonitor resource to scrape those metrics from the already enabled OADP service endpoint in the openshift-adp namespace.

For more information about setting up the monitoring stack, see Configuring user workload monitoring.

Prerequisites

• You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.
• You have created a cluster monitoring config map.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace by using the following command:
   terminal

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

2. Add or enable the enableUserWorkload option in the data section’s config.yaml field by using the following command:
   yaml

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       enableUserWorkload: true
   metadata:
   # ...

   where:

   enableUserWorkload

   Add this option or set to true.

3. Wait a short period to verify the User Workload Monitoring Setup by checking that the following components are up and running in the openshift-user-workload-monitoring namespace:
   terminal

   $ oc get pods -n openshift-user-workload-monitoring

   terminal

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-6844b4b99c-b57j9   2/2     Running   0          43s
   prometheus-user-workload-0             5/5     Running   0          32s
   prometheus-user-workload-1             5/5     Running   0          32s
   thanos-ruler-user-workload-0           3/3     Running   0          32s
   thanos-ruler-user-workload-1           3/3     Running   0          32s

4. Verify the existence of the user-workload-monitoring-config ConfigMap in the openshift-user-workload-monitoring. If it exists, skip the remaining steps in this procedure.
   terminal

   $ oc get configmap user-workload-monitoring-config -n openshift-user-workload-monitoring

   terminal

   Error from server (NotFound): configmaps "user-workload-monitoring-config" not found

5. Create a user-workload-monitoring-config ConfigMap object for the User Workload Monitoring, and save it under the 2_configure_user_workload_monitoring.yaml file name:
   yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |

6. Apply the 2_configure_user_workload_monitoring.yaml file by using the following command:
   terminal

   $ oc apply -f 2_configure_user_workload_monitoring.yaml
   configmap/user-workload-monitoring-config created

Creating OADP service monitor¶

Create a ServiceMonitor resource to scrape Velero metrics from the OADP service endpoint. This helps you collect metrics for monitoring backup and restore operations in the OpenShift Container Platform monitoring stack.

OADP provides an openshift-adp-velero-metrics-svc service. The user workload monitoring service monitor must use the openshift-adp-velero-metrics-svc service.

Procedure

1. Ensure that the openshift-adp-velero-metrics-svc service exists. It should contain app.kubernetes.io/name=velero label, which is used as selector for the ServiceMonitor object.
   terminal

   $ oc get svc -n openshift-adp -l app.kubernetes.io/name=velero

   Example output

   NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
   openshift-adp-velero-metrics-svc   ClusterIP   172.30.38.244   <none>        8085/TCP   1h

2. Create a ServiceMonitor YAML file that matches the existing service label, and save the file as 3_create_oadp_service_monitor.yaml. The service monitor is created in the openshift-adp namespace which has the openshift-adp-velero-metrics-svc service.
   yaml

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       app: oadp-service-monitor
     name: oadp-service-monitor
     namespace: openshift-adp
   spec:
     endpoints:
     - interval: 30s
       path: /metrics
       targetPort: 8085
       scheme: http
     selector:
       matchLabels:
         app.kubernetes.io/name: "velero"

3. Apply the 3_create_oadp_service_monitor.yaml file:
   terminal

   $ oc apply -f 3_create_oadp_service_monitor.yaml

   Example output

   servicemonitor.monitoring.coreos.com/oadp-service-monitor created

Verification

• Confirm that the new service monitor is in an Up state by using the Administrator perspective of the OpenShift Container Platform web console. Wait a few minutes for the service monitor to reach the Up state.
  1. Navigate to the Observe → Targets page.
  2. Ensure the Filter is unselected or that the User source is selected and type openshift-adp in the Text search field.
  3. Verify that the status for the Status for the service monitor is Up.

     Figure 1. OADP metrics targets

Creating an alerting rule¶

Create a PrometheusRule resource to configure alerting rules for OADP backup operations. This helps you receive notifications when backup failures or other issues occur in your environment.

The OpenShift Container Platform monitoring stack receives alerts configured by using alerting rules. To create an alerting rule for the OADP project, use one of the metrics scraped with the user workload monitoring.

Procedure

1. Create a PrometheusRule YAML file with the sample OADPBackupFailing alert and save it as 4_create_oadp_alert_rule.yaml:
   yaml

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: sample-oadp-alert
     namespace: openshift-adp
   spec:
     groups:
     - name: sample-oadp-backup-alert
       rules:
       - alert: OADPBackupFailing
         annotations:
           description: 'OADP had {{$value | humanize}} backup failures over the last 2 hours.'
           summary: OADP has issues creating backups
         expr: |
           increase(velero_backup_failure_total{job="openshift-adp-velero-metrics-svc"}[2h]) > 0
         for: 5m
         labels:
           severity: warning

   In this sample, the Alert displays under the following conditions:

   • During the last 2 hours, the number of new failing backups was greater than 0 and the state persisted for at least 5 minutes.
   • If the time of the first increase is less than 5 minutes, the Alert is in a Pending state, after which it turns into a Firing state.
2. Apply the 4_create_oadp_alert_rule.yaml file, which creates the PrometheusRule object in the openshift-adp namespace:
   terminal

   $ oc apply -f 4_create_oadp_alert_rule.yaml

   Example output

   prometheusrule.monitoring.coreos.com/sample-oadp-alert created

Verification

• After the Alert is triggered, you can view it in the following ways:
  • In the Developer perspective, select the Observe menu.
  • In the Administrator perspective under the Observe → Alerting menu, select User in the Filter box. Otherwise, by default only the Platform Alerts are displayed.

    Figure 2. OADP backup failing alert

List of available metrics¶

Review the following table for a list of Velero metrics provided by OADP together with their Types:

Velero metrics

Viewing metrics using the Observe UI¶

Review metrics in the OpenShift Container Platform web console from the Administrator or Developer perspective, which must have access to the openshift-adp project.

Procedure

• Navigate to the Observe → Metrics page:
  • If you are using the Developer perspective, follow these steps:
    1. Select Custom query, or click the Show PromQL link.
    2. Type the query and click Enter.
  • If you are using the Administrator perspective, type the expression in the text field and select Run Queries.

    Figure 3. OADP metrics query