--- title: DNS Operator in OpenShift Container Platform --- # DNS Operator in OpenShift Container Platform {#dns-operator} In OpenShift Container Platform, the DNS Operator deploys and manages a CoreDNS instance to provide a name resolution service to pods inside the cluster, enables DNS-based Kubernetes Service discovery, and resolves internal `cluster.local` names. ## Checking the status of the DNS Operator {#nw-dns-operator_dns-operator} You can check the DNS Operator deployment and cluster operator status. The DNS Operator is deployed during installation with a `Deployment` object. The DNS Operator implements the `dns` API from the `operator.openshift.io` API group. The Operator deploys CoreDNS using a daemon set, creates a service for the daemon set, and configures the kubelet to instruct pods to use the CoreDNS service IP address for name resolution. **Procedure** 1. Use the `oc get` command to view the deployment status: ```terminal $ oc get -n openshift-dns-operator deployment/dns-operator ``` ```terminal {title="Example output"} NAME READY UP-TO-DATE AVAILABLE AGE dns-operator 1/1 1 1 23h ``` 2. Use the `oc get` command to view the state of the DNS Operator: ```terminal $ oc get clusteroperator/dns ``` ```terminal {title="Example output"} NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE dns 4.1.15-0.11 True False False 92m ``` `AVAILABLE`, `PROGRESSING`, and `DEGRADED` provide information about the status of the Operator. `AVAILABLE` is `True` when at least 1 pod from the CoreDNS daemon set reports an `Available` status condition, and the DNS service has a cluster IP address. ## View the default DNS {#nw-dns-view_dns-operator} View the default DNS resource and cluster DNS settings to verify the DNS configuration or troubleshoot DNS issues. Every new OpenShift Container Platform installation has a `dns.operator` named `default`. **Procedure** 1. Use the `oc describe` command to view the default `dns`: ```terminal $ oc describe dns.operator/default ``` ```terminal {title="Example output"} Name: default Namespace: Labels: Annotations: API Version: operator.openshift.io/v1 Kind: DNS ... Status: Cluster Domain: cluster.local Cluster IP: 172.30.0.10 ... ``` where: `Status.Cluster Domain` : Specifiecs the base DNS domain used to construct fully qualified pod and service domain names. `Status.Cluster IP` : Specifies the address that pods query for name resolution. The IP is defined as the 10th address in the service CIDR range. 2. To find the service CIDR range, such as `172.30.0.0/16`, of your cluster, use the `oc get` command: ```terminal $ oc get networks.config/cluster -o jsonpath='{$.status.serviceNetwork}' ``` ## Using DNS forwarding {#nw-dns-forward_dns-operator} Configure DNS forwarding servers and upstream resolvers for the cluster. You can use DNS forwarding to override the default forwarding configuration in the `/etc/resolv.conf` file in the following ways: - Specify name servers (`spec.servers`) for every zone. If the forwarded zone is the ingress domain managed by OpenShift Container Platform, then the upstream name server must be authorized for the domain. - Provide a list of upstream DNS servers (`spec.upstreamResolvers`). - Change the default forwarding policy. A DNS forwarding configuration for the default domain can have both the default servers specified in the `/etc/resolv.conf` file and the upstream DNS servers. > [!IMPORTANT] > During pod creation, Kubernetes uses the `/etc/resolv.conf` file that exists on a node. If you modify the `/etc/resolv.conf` file on a host node, the changes do not propagate to the `/etc/resolv.conf` file that exists in a container. You must re-create the container for changes to take effect. **Procedure** - Modify the DNS Operator object named `default`: ```terminal $ oc edit dns.operator/default ``` After you issue the previous command, the Operator creates and updates the config map named `dns-default` with additional server configuration blocks based on `spec.servers`. If none of the servers have a zone that matches the query, then name resolution falls back to the upstream DNS servers. ```yaml {title="Configuring DNS forwarding"} apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: cache: negativeTTL: 0s positiveTTL: 0s logLevel: Normal nodePlacement: {} operatorLogLevel: Normal servers: - name: example-server zones: - example.com forwardPlugin: policy: Random upstreams: - 1.1.1.1 - 2.2.2.2:5353 upstreamResolvers: policy: Random protocolStrategy: "" transportConfig: {} upstreams: - type: SystemResolvConf - type: Network address: 1.2.3.4 port: 53 status: clusterDomain: cluster.local clusterIP: x.y.z.10 conditions: ... ``` where: `spec.servers.name` : Must comply with the `rfc6335` service name syntax. `spec.servers.zones` : Must conform to the `rfc1123` subdomain syntax. The cluster domain `cluster.local` is invalid for `zones`. `spec.servers.forwardPlugin.policy` : Specifies the upstream selection policy. Defaults to `Random`; allowed values are `RoundRobin` and `Sequential`. `spec.servers.forwardPlugin.upstreams` : Must provide no more than 15 `upstreams` entries per `forwardPlugin`. `spec.upstreamResolvers.upstreams` : Specifies an `upstreamResolvers` to override the default forwarding policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. You can use this field when you need custom upstream resolvers; otherwise queries use the servers declared in `/etc/resolv.conf`. `spec.upstreamResolvers.policy` : Specifies the upstream selection order. Defaults to `Sequential`; allowed values are `Random`, `RoundRobin`, and `Sequential`. `spec.upstreamResolvers.protocolStrategy` : Specify `TCP` to force the protocol to use for upstream DNS requests, even if the request uses UDP. Valid values are `TCP` and omitted. When omitted, the platform chooses a default, normally the protocol of the original client request. `spec.upstreamResolvers.transportConfig` : Specifies the transport type, server name, and optional custom CA or CA bundle to use when forwarding DNS requests to an upstream resolver. `spec.upstreamResolvers.upstreams.type` : Specifies two types of `upstreams`: `SystemResolvConf` or `Network`. `SystemResolvConf` configures the upstream to use `/etc/resolv.conf` and `Network` defines a `Networkresolver`. You can specify one or both. `spec.upstreamResolvers.upstreams.address` : Specifies a valid IPv4 or IPv6 address when type is `Network`. `spec.upstreamResolvers.upstreams.port` : Specifies an optional field to provide a port number. Valid values are between `1` and `65535`; defaults to 853 when omitted. **Additional resources** {._additional-resources} - [CoreDNS forward documentation](https://coredns.io/plugins/forward/) ## Checking DNS Operator status {#nw-dns-operator-status_dns-operator} You can inspect the status and view the details of the DNS Operator by using the `oc describe` command. **Procedure** - View the status of the DNS Operator: ```terminal $ oc describe clusteroperators/dns ``` Though the messages and spelling might vary in a specific release, the expected status output looks like: ```terminal Status: Conditions: Last Transition Time: Message: DNS "default" is available. Reason: AsExpected Status: True Type: Available Last Transition Time: Message: Desired and current number of DNSes are equal Reason: AsExpected Status: False Type: Progressing Last Transition Time: Reason: DNSNotDegraded Status: False Type: Degraded Last Transition Time: Message: DNS default is upgradeable: DNS Operator can be upgraded Reason: DNSUpgradeable Status: True Type: Upgradeable ``` ## Viewing DNS Operator logs {#nw-dns-operator-logs_dns-operator} You can view DNS Operator logs to troubleshoot DNS issues, verify configuration changes, and monitor activity by using the by using the `oc logs` command. **Procedure** - View the logs of the DNS Operator by running the following command: ```terminal $ oc logs -n openshift-dns-operator deployment/dns-operator -c dns-operator ``` ## Setting the CoreDNS log level {#nw-dns-loglevel_dns-operator} Set CoreDNS log levels to control the detail of DNS error logging. Log levels for CoreDNS and the CoreDNS Operator are set by using different methods. You can configure the CoreDNS log level to determine the amount of detail in logged error messages. The valid values for CoreDNS log level are `Normal`, `Debug`, and `Trace`. The default `logLevel` is `Normal`. > [!NOTE] > The CoreDNS error log level is always enabled. The following log level settings report different error responses: > > - `logLevel`: `Normal` enables the "errors" class: `log . { class error }`. > - `logLevel`: `Debug` enables the "denial" class: `log . { class denial error }`. > - `logLevel`: `Trace` enables the "all" class: `log . { class all }`. **Procedure** - To set `logLevel` to `Debug`, enter the following command: ```terminal $ oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Debug"}}' --type=merge ``` - To set `logLevel` to `Trace`, enter the following command: ```terminal $ oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Trace"}}' --type=merge ``` **Verification** - To ensure the desired log level was set, check the config map: ```terminal $ oc get configmap/dns-default -n openshift-dns -o yaml ``` For example, after setting the `logLevel` to `Trace`, you should see this stanza in each server block: ```yaml errors log . { class all } ``` ## Viewing the CoreDNS logs {#nw-dns-viewlog_dns-operator} You can view CoreDNS pod logs to troubleshoot DNS issues by using the `oc logs` command. **Procedure** - View the logs of a specific CoreDNS pod by entering the following command: ```terminal $ oc -n openshift-dns logs -c dns ``` - Follow the logs of all CoreDNS pods by entering the following command: ```terminal $ oc -n openshift-dns logs -c dns -l dns.operator.openshift.io/daemonset-dns=default -f --max-log-requests= (1) ``` - ``: Specifies the number of DNS pods to stream logs from. The maximum is 6. ## Setting the CoreDNS Operator log level {#nw-dns-operatorloglevel_dns-operator} You can configure the Operator log level to quickly track down OpenShift DNS issues. The valid values for `operatorLogLevel` are `Normal`, `Debug`, and `Trace`. `Trace` has the most detailed information. The default `operatorlogLevel` is `Normal`. There are seven logging levels for Operator issues: Trace, Debug, Info, Warning, Error, Unrecoverable, and Panic. After the logging level is set, log entries with that severity or anything above it will be logged. - `operatorLogLevel: "Normal"` sets `logrus.SetLogLevel("Info")`. - `operatorLogLevel: "Debug"` sets `logrus.SetLogLevel("Debug")`. - `operatorLogLevel: "Trace"` sets `logrus.SetLogLevel("Trace")`. **Procedure** - To set `operatorLogLevel` to `Debug`, enter the following command: ```terminal $ oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Debug"}}' --type=merge ``` - To set `operatorLogLevel` to `Trace`, enter the following command: ```terminal $ oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Trace"}}' --type=merge ``` **Verification** 1. To review the resulting change, enter the following command: ```terminal $ oc get dnses.operator -A -oyaml ``` You should see two log level entries. The `operatorLogLevel` applies to OpenShift DNS Operator issues, and the `logLevel` applies to the daemonset of CoreDNS pods: ```yaml logLevel: Trace operatorLogLevel: Debug ``` 2. To review the logs for the daemonset, enter the following command: ```terminal $ oc logs -n openshift-dns ds/dns-default ``` ## Tuning the CoreDNS cache {#nw-dns-cache-tuning_dns-operator} To reduce the load on upstream DNS resolvers, you can tune the CoreDNS cache by adjusting the duration of positive and negative caching. This process involves modifying the time-to-live (TTL) values within the DNS Operator object to control how long query responses are stored. For CoreDNS, you can configure the maximum duration of both successful or unsuccessful caching, also known respectively as positive or negative caching. Tuning the cache duration of DNS query responses can reduce the load for any upstream DNS resolvers. You can shorten the TTL of the DNS record by setting a lower positive cache. You cannot increase the TTL on the DNS record by setting a higher positive cache. The maximum cache is the lower of the TTL of the DNS record or the positive cache. > [!WARNING] > Setting TTL fields to low values could lead to an increased load on the cluster, any upstream resolvers, or both. **Procedure** 1. Edit the DNS Operator object named `default` by running the following command: ```terminal $ oc edit dns.operator.openshift.io/default ``` 2. Modify the time-to-live (TTL) caching values: ```yaml {title="Configuring DNS caching"} apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: cache: positiveTTL: 1h negativeTTL: 0.5h10m ``` where: `spec.cache.positiveTTL` : Specifies a string value that is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be `0s` and the cluster uses the internal default value of `900s` as a fallback. `spec.cache.negativeTTL` : Specifies a string value that is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be `0s` and the cluster uses the internal default value of `30s` as a fallback. **Verification** 1. To review the change, look at the config map again by running the following command: ```terminal $ oc get configmap/dns-default -n openshift-dns -o yaml ``` 2. Verify that you see entries that look like the following example: ```yaml cache 3600 { denial 9984 2400 } ``` **Additional resources** {._additional-resources} - [CoreDNS cache](https://coredns.io/plugins/cache/) ### Changing the DNS Operator managementState {#nw-dns-operator-managementState_dns-operator} You can change from the default `Managed` state to `Unmanaged` to stop the DNS Operator from managing its resources in order to apply a workaround or test a configuration change. The DNS Operator manages the CoreDNS component to provide a name resolution service for pods and services in the cluster. The `managementState` of the DNS Operator is set to `Managed` by default, which means that the DNS Operator is actively managing its resources. You can change it to `Unmanaged`, which means the DNS Operator is not managing its resources. The following are use cases for changing the DNS Operator `managementState`: - You are a developer and want to test a configuration change to see if it fixes an issue in CoreDNS. You can stop the DNS Operator from overwriting the configuration change by setting the `managementState` to `Unmanaged`. - You are a cluster administrator and have reported an issue with CoreDNS, but need to apply a workaround until the issue is fixed. You can set the `managementState` field of the DNS Operator to `Unmanaged` to apply the workaround. > [!NOTE] > You cannot upgrade while the `managementState` is set to `Unmanaged`. **Procedure** 1. Change `managementState` to `Unmanaged` in the DNS Operator by running the following command: ```terminal oc patch dns.operator.openshift.io default --type merge --patch '{"spec":{"managementState":"Unmanaged"}}' ``` 2. Review `managementState` of the DNS Operator by using the `jsonpath` command-line JSON parser: ```terminal $ oc get dns.operator.openshift.io default -ojsonpath='{.spec.managementState}' ``` ### Controlling DNS pod placement {#nw-controlling-dns-pod-placement_dns-operator} Control where CoreDNS and node-resolver pods run by using taints, tolerations, and selectors. The DNS Operator has two daemon sets: one for CoreDNS called `dns-default` and one for managing the `/etc/hosts` file called `node-resolver`. You can assign and run CoreDNS pods on specified nodes. For example, if the cluster administrator has configured security policies that prohibit communication between pairs of nodes, you can configure CoreDNS pods to run on a restricted set of nodes. DNS service is available to all pods if the following circumstances are true: - DNS pods are running on some nodes in the cluster. - The nodes on which DNS pods are not running have network connectivity to nodes on which DNS pods are running, The `node-resolver` daemon set must run on every node host because it adds an entry for the cluster image registry to support pulling images. The `node-resolver` pods have only one job: to look up the `image-registry.openshift-image-registry.svc` service’s cluster IP address and add it to `/etc/hosts` on the node host so that the container runtime can resolve the service name. As a cluster administrator, you can use a custom node selector to configure the daemon set for CoreDNS to run or not run on certain nodes. **Prerequisites** - You installed the `oc` CLI. - You are logged in to the cluster as a user with `cluster-admin` privileges. - Your DNS Operator `managementState` is set to `Managed`. **Procedure** - To allow the daemon set for CoreDNS to run on certain nodes, configure a taint and toleration: 1. Set a taint on the nodes that you want to control DNS pod placement by entering the following command: ```terminal $ oc adm taint nodes dns-only=abc:NoExecute ``` - Replace `` with the actual name of the node. 2. Modify the DNS Operator object named `default` to include the corresponding toleration by entering the following command: ```terminal $ oc edit dns.operator/default ``` 3. Specify a taint key and a toleration for the taint. The following toleration matches the taint set on the nodes. ```yaml apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: nodePlacement: tolerations: - effect: NoExecute key: "dns-only" operator: Equal value: abc tolerationSeconds: 3600 ``` - If the `key` field is set to `dns-only`, it can be tolerated indefinitely. - The `tolerationSeconds` field is optional. 4. Optional: To specify node placement using a node selector, modify the default DNS Operator: 1. Edit the DNS Operator object named `default` to include a node selector: ```yaml apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: nodePlacement: nodeSelector: node-role.kubernetes.io/control-plane: "" ``` - The `spec.nodePlacement.nodeSelector` field in the example ensures that the CoreDNS pods run only on control plane nodes. ### Configuring DNS forwarding with TLS {#configuring-dns-forwarding-with-tls_dns-operator} Configure DNS forwarding with TLS to secure queries to upstream resolvers. When working in a highly regulated environment, you might need the ability to secure DNS traffic when forwarding requests to upstream resolvers so that you can ensure additional DNS traffic and data privacy. Be aware that CoreDNS caches forwarded connections for 10 seconds. CoreDNS will hold a TCP connection open for those 10 seconds if no request is issued. > [!NOTE] > With large clusters, ensure that your DNS server is aware that it might get many new connections to hold open because you can initiate a connection per node. Set up your DNS hierarchy accordingly to avoid performance issues. **Procedure** 1. Modify the DNS Operator object named `default`: ```terminal $ oc edit dns.operator/default ``` Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries. ```yaml {title="Configuring DNS forwarding with TLS"} apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: servers: - name: example_server zones: - example.com forwardPlugin: transportConfig: transport: TLS tls: caBundle: name: mycacert serverName: dnstls.example.com policy: Random upstreams: - 1.1.1.1 - 2.2.2.2:5353 upstreamResolvers: transportConfig: transport: TLS tls: caBundle: name: mycacert serverName: dnstls.example.com upstreams: - type: Network address: 1.2.3.4 port: 53 ``` where: `spec.servers.name` : Must comply with the `rfc6335` service name syntax. `spec.servers.zones` : Must conform to the `rfc1123` subdomain syntax. The cluster domain, `cluster.local`, is invalid for `zones`. `spec.servers.forwardPlugin.transportConfig.transport` : Must be set to `TLS` when configuring TLS forwarding. `spec.servers.forwardPlugin.transportConfig.tls.serverName` : Must be set to the server name indication (SNI) server name used to validate the upstream TLS certificate. `spec.servers.forwardPlugin.policy` : Specifies the upstream selection policy. Defaults to `Random`; valid values are `RoundRobin` and `Sequential`. `spec.servers.forwardPlugin.upstreams` : Must provide upstream resolvers; maximum 15 entries per `forwardPlugin`. `spec.upstreamResolvers.upstreams` : Specifies an optional field to override the default policy for the default domain. Use the `Network` type only when TLS is enabled and provide an IP address. If omitted, queries use `/etc/resolv.conf`. `spec.upstreamResolvers.upstreams.address` : Must be a valid IPv4 or IPv6 address. `spec.upstreamResolvers.upstreams.port` : Specifies an optional field to provide a port number. Valid values are between `1` and `65535`; defaults to 853 when omitted. > [!NOTE] > If `servers` is undefined or invalid, the config map only contains the default server. **Verification** 1. View the config map: ```terminal $ oc get configmap/dns-default -n openshift-dns -o yaml ``` ```yaml {title="Sample DNS ConfigMap based on TLS forwarding example"} apiVersion: v1 data: Corefile: | example.com:5353 { forward . 1.1.1.1 2.2.2.2:5353 } bar.com:5353 example.com:5353 { forward . 3.3.3.3 4.4.4.4:5454 } .:5353 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf 1.2.3.4:53 { policy Random } cache 30 reload } kind: ConfigMap metadata: labels: dns.operator.openshift.io/owning-dns: default name: dns-default namespace: openshift-dns ``` - The `data.Corefile` key contains the Corefile configuration for the DNS server. Changes to the `forwardPlugin` triggers a rolling update of the CoreDNS daemon set. **Additional resources** {._additional-resources} - [CoreDNS forward documentation](https://coredns.io/plugins/forward/)