--- title: Exposing custom metrics for virtual machines --- # Exposing custom metrics for virtual machines {#virt-exposing-custom-metrics-for-vms} Monitor core platform components by using the OpenShift Container Platform monitoring stack based on the Prometheus monitoring system. Additionally, enable monitoring for user-defined projects by using the CLI and query custom metrics exposed for virtual machines through the `node-exporter` service. ## Configuring the node exporter service {#virt-configuring-node-exporter-service_virt-exposing-custom-metrics-for-vms} The node-exporter agent is deployed on every virtual machine in the cluster from which you want to collect metrics. Configure the node-exporter agent as a service to expose internal metrics and processes that are associated with virtual machines. **Prerequisites** - Install the OpenShift CLI (`oc`). - Log in to the cluster as a user with `cluster-admin` privileges. - Create the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project. - Configure the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project by setting `enableUserWorkload` to `true`. **Procedure** 1. Create the `Service` YAML file. In the following example, the file is called `node-exporter-service.yaml`. ```yaml kind: Service apiVersion: v1 metadata: name: node-exporter-service namespace: dynamation labels: servicetype: metrics spec: ports: - name: exmet protocol: TCP port: 9100 targetPort: 9100 type: ClusterIP selector: monitor: metrics ``` - `metadata.name` defines the node-exporter service that exposes the metrics from the virtual machines. - `metadata.namespace` defines the namespace where the service is created. - `metadata.labels.servicetype` defines the label for the service. The `ServiceMonitor` uses this label to match this service. - `spec.ports.name` defines the name given to the port that exposes metrics on port 9100 for the `ClusterIP` service. - `spec.ports.port` defines the target port used by `node-exporter-service` to listen for requests. - `spec.ports.targetPort` defines the TCP port number of the virtual machine that is configured with the `monitor` label. - `spec.selector.monitor` defines the label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label `monitor` and a value of `metrics` will be matched. 2. Create the node-exporter service: ```terminal $ oc create -f node-exporter-service.yaml ``` ## Configuring a virtual machine with the node exporter service {#virt-configuring-vm-with-node-exporter-service_virt-exposing-custom-metrics-for-vms} Download the `node-exporter` file on to the virtual machine. Then, create a `systemd` service that runs the node-exporter service when the virtual machine boots. **Prerequisites** - The pods for the component are running in the `openshift-user-workload-monitoring` project. - Grant the `monitoring-edit` role to users who need to monitor this user-defined project. **Procedure** 1. Log on to the virtual machine. 2. Download the `node-exporter` file on to the virtual machine by using the directory path that applies to the version of `node-exporter` file. ```terminal $ wget https://github.com/prometheus/node_exporter/releases/download//node_exporter-.linux-.tar.gz ``` 3. Extract the executable and place it in the `/usr/bin` directory. ```terminal $ sudo tar xvf node_exporter-.linux-.tar.gz \ --directory /usr/bin --strip 1 "*/node_exporter" ``` 4. Create a `node_exporter.service` file in this directory path: `/etc/systemd/system`. This `systemd` service file runs the node-exporter service when the virtual machine reboots. ```terminal [Unit] Description=Prometheus Metrics Exporter After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=root ExecStart=/usr/bin/node_exporter [Install] WantedBy=multi-user.target ``` 5. Enable and start the `systemd` service. ```terminal $ sudo systemctl enable node_exporter.service ``` ```terminal $ sudo systemctl start node_exporter.service ``` **Verification** - Verify that the node-exporter agent is reporting metrics from the virtual machine. ```terminal $ curl http://localhost:9100/metrics ``` Example output: ```terminal go_gc_duration_seconds{quantile="0"} 1.5244e-05 go_gc_duration_seconds{quantile="0.25"} 3.0449e-05 go_gc_duration_seconds{quantile="0.5"} 3.7913e-05 ``` ## Creating a custom monitoring label for virtual machines {#virt-creating-custom-monitoring-label-for-vms_virt-exposing-custom-metrics-for-vms} To enable queries to multiple virtual machines from a single service, you can add a custom label in the virtual machine’s YAML file. **Prerequisites** - Install the OpenShift CLI (`oc`). - Log in as a user with `cluster-admin` privileges. - Access to the web console for stop and restart a virtual machine. **Procedure** 1. Edit the `template` spec of your virtual machine configuration file. In this example, the label `monitor` has the value `metrics`. ```yaml spec: template: metadata: labels: monitor: metrics ``` 2. Stop and restart the virtual machine to create a new pod with the label name given to the `monitor` label. ### Querying the node-exporter service for metrics {#virt-querying-the-node-exporter-service-for-metrics-_virt-exposing-custom-metrics-for-vms} Metrics are exposed for virtual machines through an HTTP service endpoint under the `/metrics` canonical name. When you query for metrics, Prometheus directly scrapes the metrics from the metrics endpoint exposed by the virtual machines and presents these metrics for viewing. **Prerequisites** - You have access to the cluster as a user with `cluster-admin` privileges or the `monitoring-edit` role. - You have enabled monitoring for the user-defined project by configuring the node-exporter service. - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Obtain the HTTP service endpoint by specifying the namespace for the service: ```terminal $ oc get service -n ``` 2. To list all available metrics for the node-exporter service, query the `metrics` resource. ```terminal $ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$" ``` Example output: ```terminal node_arp_entries{device="eth0"} 1 node_boot_time_seconds 1.643153218e+09 node_context_switches_total 4.4938158e+07 node_cooling_device_cur_state{name="0",type="Processor"} 0 node_cooling_device_max_state{name="0",type="Processor"} 0 node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0 node_cpu_guest_seconds_total{cpu="0",mode="user"} 0 node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06 node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61 node_cpu_seconds_total{cpu="0",mode="irq"} 233.91 node_cpu_seconds_total{cpu="0",mode="nice"} 551.47 node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3 node_cpu_seconds_total{cpu="0",mode="steal"} 86.12 node_cpu_seconds_total{cpu="0",mode="system"} 464.15 node_cpu_seconds_total{cpu="0",mode="user"} 1075.2 node_disk_discard_time_seconds_total{device="vda"} 0 node_disk_discard_time_seconds_total{device="vdb"} 0 node_disk_discarded_sectors_total{device="vda"} 0 node_disk_discarded_sectors_total{device="vdb"} 0 node_disk_discards_completed_total{device="vda"} 0 node_disk_discards_completed_total{device="vdb"} 0 node_disk_discards_merged_total{device="vda"} 0 node_disk_discards_merged_total{device="vdb"} 0 node_disk_info{device="vda",major="252",minor="0"} 1 node_disk_info{device="vdb",major="252",minor="16"} 1 node_disk_io_now{device="vda"} 0 node_disk_io_now{device="vdb"} 0 node_disk_io_time_seconds_total{device="vda"} 174 node_disk_io_time_seconds_total{device="vdb"} 0.054 node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003 node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039 node_disk_read_bytes_total{device="vda"} 3.71867136e+08 node_disk_read_bytes_total{device="vdb"} 366592 node_disk_read_time_seconds_total{device="vda"} 19.128 node_disk_read_time_seconds_total{device="vdb"} 0.039 node_disk_reads_completed_total{device="vda"} 5619 node_disk_reads_completed_total{device="vdb"} 96 node_disk_reads_merged_total{device="vda"} 5 node_disk_reads_merged_total{device="vdb"} 0 node_disk_write_time_seconds_total{device="vda"} 240.66400000000002 node_disk_write_time_seconds_total{device="vdb"} 0 node_disk_writes_completed_total{device="vda"} 71584 node_disk_writes_completed_total{device="vdb"} 0 node_disk_writes_merged_total{device="vda"} 19761 node_disk_writes_merged_total{device="vdb"} 0 node_disk_written_bytes_total{device="vda"} 2.007924224e+09 node_disk_written_bytes_total{device="vdb"} 0 ``` ## Creating a ServiceMonitor resource for the node exporter service {#virt-creating-servicemonitor-resource-for-node-exporter_virt-exposing-custom-metrics-for-vms} You can use a Prometheus client library and scrape metrics from the `/metrics` endpoint to access and view the metrics exposed by the node-exporter service. Use a `ServiceMonitor` custom resource definition (CRD) to monitor the node exporter service. **Prerequisites** - You have access to the cluster as a user with `cluster-admin` privileges or the `monitoring-edit` role. - You have enabled monitoring for the user-defined project by configuring the node-exporter service. - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Create a YAML file for the `ServiceMonitor` resource configuration. In this example, the service monitor matches any service with the label `metrics` and queries the `exmet` port every 30 seconds. ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: node-exporter-metrics-monitor name: node-exporter-metrics-monitor namespace: dynamation spec: endpoints: - interval: 30s port: exmet scheme: http selector: matchLabels: servicetype: metrics ``` - `metadata.name` defines the name of the `ServiceMonitor`. - `metadata.namespace` defines the namespace where the `ServiceMonitor` is created. - `spec.endpoints.interval` defines the interval at which the port will be queried. - `spec.endpoints.port` defines the name of the port that is queried every 30 seconds 2. Create the `ServiceMonitor` configuration for the node-exporter service. ```terminal $ oc create -f node-exporter-metrics-monitor.yaml ``` ### Accessing the node exporter service outside the cluster {#virt-accessing-node-exporter-outside-cluster_virt-exposing-custom-metrics-for-vms} You can access the node-exporter service outside the cluster and view the exposed metrics. **Prerequisites** - You have access to the cluster as a user with `cluster-admin` privileges or the `monitoring-edit` role. - You have enabled monitoring for the user-defined project by configuring the node-exporter service. - You have installed the OpenShift CLI (`oc`). **Procedure** 1. Expose the node-exporter service. ```terminal $ oc expose service -n ``` 2. Obtain the FQDN (Fully Qualified Domain Name) for the route. ```terminal $ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host ``` Example output: ```terminal NAME DNS node-exporter-service node-exporter-service-dynamation.apps.cluster.example.org ``` 3. Use the `curl` command to display metrics for the node-exporter service. ```terminal $ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics ``` Example output: ```terminal go_gc_duration_seconds{quantile="0"} 1.5382e-05 go_gc_duration_seconds{quantile="0.25"} 3.1163e-05 go_gc_duration_seconds{quantile="0.5"} 3.8546e-05 go_gc_duration_seconds{quantile="0.75"} 4.9139e-05 go_gc_duration_seconds{quantile="1"} 0.000189423 ``` **Additional resources** {._additional-resources} - [Core platform monitoring first steps](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/getting_started/core-platform-monitoring-first-steps) - [Enabling monitoring for user-defined projects](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/configuring_user_workload_monitoring/preparing-to-configure-the-monitoring-stack-uwm#enabling-monitoring-for-user-defined-projects-uwm_preparing-to-configure-the-monitoring-stack-uwm) - [Accessing metrics as a developer](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/accessing_metrics/accessing-metrics-as-a-developer) - [Reviewing monitoring dashboards as a developer](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/latest/html/accessing_metrics/accessing-metrics-as-a-developer#reviewing-monitoring-dashboards-developer_accessing-metrics-as-a-developer) - [Monitoring application health by using health checks](/openshift-docs-markdown/applications/application-health#application-health) - [Creating and using config maps](/openshift-docs-markdown/nodes/pods/nodes-pods-configmaps#nodes-pods-configmap-overview_configmaps) - [Controlling virtual machine states](/openshift-docs-markdown/virt/managing_vms/virt-controlling-vm-states#virt-controlling-vm-states)