--- title: Kubernetes NMState Operator --- # Kubernetes NMState Operator {#k8s-nmstate-about-the-k8s-nmstate-operator} The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across the OpenShift Container Platform cluster nodes with NMState. The Kubernetes NMState Operator provides users with functionality to configure various network interface types, DNS, and routing on cluster nodes. Additionally, the daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server. > [!IMPORTANT] > Red Hat supports the Kubernetes NMState Operator in production environments on bare-metal, IBM Power(R), IBM Z(R), IBM(R) LinuxONE, VMware vSphere, and Red Hat OpenStack Platform (RHOSP) installations. > > Red Hat support exists for using the Kubernetes NMState Operator on Microsoft Azure but in a limited capacity. Support is limited to configuring DNS servers on your system as a postinstallation task. Before you can use NMState with OpenShift Container Platform, you must install the Kubernetes NMState Operator. After you install the Kubernetes NMState Operator, you can complete the following tasks: - Observing and updating the node network state and configuration - Creating a manifest object that includes a customized `br-ex` bridge > [!IMPORTANT] > The Kubernetes NMState Operator updates the network configuration of a secondary NIC. Do not use the Operator to update the primary NIC network configuration or the `br-ex` bridge on most on-premise networks. Applying a `NodeNetworkConfigurationPolicy` CR to the primary NIC or `br-ex` bridge can result in complete network loss on the affected node. This configuration requires manual recovery of the network configuration. > > For a bare-metal platform only, the Kubernetes NMState Operator can update the `br-ex` bridge network configuration. This update is supported only if you set the `br-ex` bridge as the interface in a machine config manifest file. To update the `br-ex` bridge as a postinstallation task, you must set the `br-ex` bridge as the interface in the NMState configuration of the `NodeNetworkConfigurationPolicy` custom resource (CR) for your cluster. For more information, see "Creating a manifest object that includes a customized br-ex bridge (Post-installation documentation)". OpenShift Container Platform uses `nmstate` to report on and configure the state of the node network. You can modify the network policy configuration by applying a single configuration manifest to the cluster. For example, you can create a Linux bridge on all nodes. Node networking is monitored and updated by the following objects: `NodeNetworkState` : Reports the state of the network on that node. `NodeNetworkConfigurationPolicy` : Describes the requested network configuration on nodes. You update the node network configuration, including adding and removing interfaces, by applying a `NodeNetworkConfigurationPolicy` CR to the cluster. `NodeNetworkConfigurationEnactment` : Reports the network policies enacted upon each node. > [!NOTE] > Do not make configuration changes to the `br-ex` bridge or its underlying interfaces as a postinstallation task. ## Installing the Kubernetes NMState Operator {#installing-the-kubernetes-nmstate-operator-cli} You can install the Kubernetes NMState Operator by using the web console or the CLI. ### Installing the Kubernetes NMState Operator by using the web console {#installing-the-kubernetes-nmstate-operator-web-console_k8s-nmstate-about-the-k8s-nmstate-operator} You can install the Kubernetes NMState Operator by using the web console. After you install the Kubernetes NMState Operator, the Operator has deployed the NMState State Controller as a daemon set across all of the cluster nodes. **Prerequisites** - You are logged in as a user with `cluster-admin` privileges. **Procedure** 1. Select **Ecosystem** → **Software Catalog**. 2. In the search field below **All Items**, enter `nmstate` and click **Enter** to search for the Kubernetes NMState Operator. 3. Click on the Kubernetes NMState Operator search result. 4. Click on **Install** to open the **Install Operator** window. 5. Click **Install** to install the Operator. 6. After the Operator finishes installing, click **View Operator**. 7. Under **Provided APIs**, click **Create Instance** to open the dialog box for creating an instance of `kubernetes-nmstate`. 8. In the **Name** field of the dialog box, ensure the name of the instance is `nmstate.` > [!NOTE] > The name restriction is a known issue. The instance is a singleton for the entire cluster. 9. Accept the default settings and click **Create** to create the instance. ### Installing the Kubernetes NMState Operator by using the CLI {#installing-the-kubernetes-nmstate-operator-CLI_k8s-nmstate-about-the-k8s-nmstate-operator} You can install the Kubernetes NMState Operator by using the OpenShift CLI (`oc)`. After it is installed, the Operator deploys the NMState State Controller as a daemon set across all of the cluster nodes to manage the node network state and configuration. **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You are logged in as a user with `cluster-admin` privileges. **Procedure** 1. Create the `nmstate` Operator namespace: ```terminal $ cat << EOF | oc apply -f - apiVersion: v1 kind: Namespace metadata: name: openshift-nmstate spec: finalizers: - kubernetes EOF ``` 2. Create the `OperatorGroup`: ```terminal $ cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-nmstate namespace: openshift-nmstate spec: targetNamespaces: - openshift-nmstate EOF ``` 3. Subscribe to the `nmstate` Operator: ```terminal $ cat << EOF| oc apply -f - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: kubernetes-nmstate-operator namespace: openshift-nmstate spec: channel: stable installPlanApproval: Automatic name: kubernetes-nmstate-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF ``` 4. Confirm the `ClusterServiceVersion` (CSV) status for the `nmstate` Operator deployment equals `Succeeded`: ```terminal $ oc get clusterserviceversion -n openshift-nmstate \ -o custom-columns=Name:.metadata.name,Phase:.status.phase ``` 5. Create an instance of the `nmstate` Operator: ```terminal $ cat << EOF | oc apply -f - apiVersion: nmstate.io/v1 kind: NMState metadata: name: nmstate EOF ``` 6. If your cluster has problems with the DNS health check probe because of DNS connectivity issues, you can add the following DNS host name configuration to the `NMState` CRD to build in health checks that can resolve these issues: ```terminal apiVersion: nmstate.io/v1 kind: NMState metadata: name: nmstate spec: probeConfiguration: dns: host: redhat.com # ... ``` 1. Apply the DNS host name configuration to your cluster network by running the following command. Ensure that you replace `` with the name of your CRD file. ```yaml $ oc apply -f .yaml ``` 2. Monitor the `nmstate` CRD until the resource reaches the `Available` condition by running the following command. Ensure that you set a value for the `--timeout` option so that if the `Available` condition is not met within this set maximum waiting time, the command times out and generates an error message. ```yaml $ oc wait --for=condition=Available nmstate/nmstate --timeout=600s ``` **Verification** 1. Verify that all pods for the NMState Operator have the `Running` status by entering the following command: ```terminal $ oc get pod -n openshift-nmstate ``` ### Viewing metrics collected by the Kubernetes NMState Operator {#viewing-stats-collected-kubernetes-nmstate-op_k8s-nmstate-about-the-k8s-nmstate-operator} The Kubernetes NMState Operator, `kubernetes-nmstate-operator`, can collect metrics from the Kubernetes components and expose them as ready-to-use metrics. The Kubernetes NMState Operator can collect metrics from the following Kubernetes components: - `kubernetes_nmstate_features_applied`, which tracks what NMState features are enabled and successfully applied to the cluster. - `kubernetes_nmstate_policies_status`, which tracks the active status of `NodeNetworkConfigurationPolicy` (NNCP) resources across the cluster. - `kubernetes_nmstate_enactments_status`, which tracks the active status of `NodeNetworkConfigurationEnactment` (NNCE) resources on a per-node basis. As a use case for viewing metrics, consider a situation where you created a `NodeNetworkConfigurationPolicy` custom resource (CR) and you want to confirm that the policy is active. > [!NOTE] > The `kubernetes_nmstate_features_applied` metrics are not an API and might change between OpenShift Container Platform versions. In the web console, the Metrics UI includes some predefined CPU, memory, bandwidth, and network packet queries for the selected project. You can run custom Prometheus Query Language (PromQL) queries for CPU, memory, bandwidth, network packet and application metrics for the project. The following example demonstrates a `NodeNetworkConfigurationPolicy` manifest example that is applied to an OpenShift Container Platform cluster: ```yaml # ... interfaces: - name: br1 type: linux-bridge state: up ipv4: enabled: true dhcp: true dhcp-custom-hostname: foo bridge: options: stp: enabled: false port: [] # ... ``` The `NodeNetworkConfigurationPolicy` manifest exposes metrics and makes them available to the Cluster Monitoring Operator (CMO). The following example shows some exposed metrics: ```terminal controller_runtime_reconcile_time_seconds_bucket{controller="nodenetworkconfigurationenactment",le="0.005"} 16 controller_runtime_reconcile_time_seconds_bucket{controller="nodenetworkconfigurationenactment",le="0.01"} 16 controller_runtime_reconcile_time_seconds_bucket{controller="nodenetworkconfigurationenactment",le="0.025"} 16 ... # HELP kubernetes_nmstate_features_applied Number of nmstate features applied labeled by its name # TYPE kubernetes_nmstate_features_applied gauge kubernetes_nmstate_features_applied{name="dhcpv4-custom-hostname"} 1 ``` **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have logged in to the web console as the administrator and installed the Kubernetes NMState Operator. - You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing metrics for. - You have enabled monitoring for user-defined projects. - You have deployed a service in a user-defined project. - You have created a `NodeNetworkConfigurationPolicy` manifest and applied it to your cluster. > [!IMPORTANT] > Starting with OpenShift Container Platform 4.19, the perspectives in the web console have unified. The **Developer** perspective is no longer enabled by default. > > All users can interact with all OpenShift Container Platform web console features. However, if you are not the cluster owner, you might need to request permission to access certain features from the cluster owner. > > You can still enable the **Developer** perspective. On the **Getting Started** pane in the web console, you can take a tour of the console, find information on setting up your cluster, view a quick start for enabling the **Developer** perspective, and follow links to explore new features and capabilities. > > See also, "Enabling the **Developer** perspective in the web console". **Procedure** 1. If you want to view the metrics from the **Developer** perspective in the OpenShift Container Platform web console, complete the following tasks: 1. Click **Observe**. 2. To view the metrics of a specific project, select the project in the **Project:** list. For example, `openshift-nmstate`. 3. Click the **Metrics** tab. 4. To visualize the metrics on the plot, select a query from the **Select query** list or create a custom PromQL query based on the selected query by selecting **Show PromQL**. > [!NOTE] > You can only run one query at a time with the developer role. 2. If you want to view the metrics in the OpenShift Container Platform web console as an administrator, complete the following tasks: 1. Click **Observe** → **Metrics**. 2. Enter `kubernetes_nmstate_features_applied` in the **Expression** field. 3. Click **Add query** and then **Run queries**. 3. To explore the visualized metrics, do any of the following tasks: 1. To zoom into the plot and change the time range, do any of the following tasks: - To visually select the time range, click and drag on the plot horizontally. - To select the time range, use the menu which is in the upper left of the console. 2. To reset the time range, select **Reset zoom**. 3. To display the output for all the queries at a specific point in time, hold the mouse cursor on the plot at that point. The query output displays in a pop-up box. ## Uninstalling the Kubernetes NMState Operator {#k8s-nmstate-uninstall-operator_k8s-nmstate-about-the-k8s-nmstate-operator} Remove the Kubernetes NMState Operator and related resources when they are no longer needed. You can use the Operator Lifecycle Manager (OLM) to uninstall the Kubernetes NMState Operator, but by design OLM does not delete any associated custom resource definitions (CRDs), custom resources (CRs), or API Services. Before you uninstall the Kubernetes NMState Operator from the `Subcription` resource used by OLM, identify what Kubernetes NMState Operator resources to delete. This identification ensures that you can delete resources without impacting your running cluster. If you need to reinstall the Kubernetes NMState Operator, see "Installing the Kubernetes NMState Operator by using the CLI" or "Installing the Kubernetes NMState Operator by using the web console". **Prerequisites** - You have installed the OpenShift CLI (`oc`). - You have installed the `jq` CLI tool. - You are logged in as a user with `cluster-admin` privileges. **Procedure** 1. Unsubscribe the Kubernetes NMState Operator from the `Subcription` resource by running the following command: ```terminal $ oc delete --namespace openshift-nmstate subscription kubernetes-nmstate-operator ``` 2. Find the `ClusterServiceVersion` (CSV) resource that associates with the Kubernetes NMState Operator: ```terminal $ oc get --namespace openshift-nmstate clusterserviceversion ``` ```terminal {title="Example output that lists a CSV resource"} NAME DISPLAY VERSION REPLACES PHASE kubernetes-nmstate-operator.v4.22.0 Kubernetes NMState Operator 4.22.0 Succeeded ``` 3. Delete the CSV resource. After you delete the file, OLM deletes certain resources, such as `RBAC`, that it created for the Operator. ```terminal $ oc delete --namespace openshift-nmstate clusterserviceversion kubernetes-nmstate-operator.v4.22.0 ``` 4. Delete the `nmstate` CR and any associated `Deployment` resources by running the following commands: ```terminal $ oc -n openshift-nmstate delete nmstate nmstate ``` ```terminal $ oc delete --all deployments --namespace=openshift-nmstate ``` 5. After you deleted the `nmstate` CR, remove the `nmstate-console-plugin` console plugin name from the `console.operator.openshift.io/cluster` CR. 1. Store the position of the `nmstate-console-plugin` entry that exists among the list of enable plugins by running the following command. The following command uses the `jq` CLI tool to store the index of the entry in an environment variable named `INDEX`: ```terminal INDEX=$(oc get console.operator.openshift.io cluster -o json | jq -r '.spec.plugins | to_entries[] | select(.value == "nmstate-console-plugin") | .key') ``` 2. Remove the `nmstate-console-plugin` entry from the `console.operator.openshift.io/cluster` CR by running the following patch command: ```terminal $ oc patch console.operator.openshift.io cluster --type=json -p "[{\"op\": \"remove\", \"path\": \"/spec/plugins/$INDEX\"}]" ``` - `INDEX` is an auxiliary variable. You can specify a different name for this variable. 6. Optional: To preserve CR instances so that you can restore them after you delete CRDs, enter the following command: ```terminal $ oc get -A nncp -o yaml > cluster-nncp.yaml ``` > [!IMPORTANT] > To reuse preserved CRs, such as NNCPs, you must uninstall the Kubernetes NMState Operator, reinstall the Kubernetes NMState Operator, and then run the following command to restore the CRs: > > ```terminal > $ oc apply -f cluster-nncp.yaml > ``` 7. Delete all the CRDs, such as `nmstates.nmstate.io`, by running the following commands: ```terminal $ oc delete crd nmstates.nmstate.io ``` ```terminal $ oc delete crd nodenetworkconfigurationenactments.nmstate.io ``` ```terminal $ oc delete crd nodenetworkstates.nmstate.io ``` ```terminal $ oc delete crd nodenetworkconfigurationpolicies.nmstate.io ``` 8. Delete the namespace: ```terminal $ oc delete namespace openshift-nmstate ``` **Additional resources** {._additional-resources} - [`nmstate`](https://nmstate.github.io/) - [Creating an interface on nodes](/openshift-docs-markdown/networking/k8s_nmstate/k8s-nmstate-updating-node-network-config#virt-creating-interface-on-nodes_k8s-nmstate-updating-node-network-config) - [Observing and updating the node network state and configuration](/openshift-docs-markdown/networking/k8s_nmstate/k8s-nmstate-updating-node-network-config#k8s-nmstate-updating-node-network-config) - [Creating a manifest object that includes a customized br-ex bridge (Installer-provisioned infrastructure)](/openshift-docs-markdown/installing/installing_bare_metal/ipi/ipi-install-installation-workflow#creating-manifest-file-customized-br-ex-bridge_ipi-install-installation-workflow) - [Creating a manifest object that includes a customized br-ex bridge (User-provisioned infrastructure)](/openshift-docs-markdown/installing/installing_bare_metal/upi/installing-bare-metal#creating-manifest-file-customized-br-ex-bridge_installing-bare-metal) - [Creating a manifest object that includes a customized br-ex bridge (Post-installation documentation)](/openshift-docs-markdown/installing/installing_bare_metal/bare-metal-postinstallation-configuration#creating-manifest-file-customized-br-ex-bridge-post_bare-metal-postinstallation-configuration)