--- title: Using CPU Manager and Topology Manager --- # Using CPU Manager and Topology Manager {#using-cpu-manager} CPU Manager manages groups of CPUs and constrains workloads to specific CPUs. CPU Manager is useful for workloads that have some of these attributes: - Require as much CPU time as possible. - Are sensitive to processor cache misses. - Are low-latency network applications. - Coordinate with other processes and benefit from sharing a single processor cache. Topology Manager collects hints from the CPU Manager, Device Manager, and other Hint Providers to align pod resources, such as CPU, SR-IOV VFs, and other device resources, for all Quality of Service (QoS) classes on the same non-uniform memory access (NUMA) node. Topology Manager uses topology information from the collected hints to decide if a pod can be accepted or rejected on a node, based on the configured Topology Manager policy and pod resources requested. Topology Manager is useful for workloads that use hardware accelerators to support latency-critical execution and high throughput parallel computation. To use Topology Manager you must configure CPU Manager with the `static` policy. ## Setting up CPU Manager {#setting_up_cpu_manager_using-cpu-manager-and-topology-manager} To configure CPU manager, create a `KubeletConfig` custom resource (CR) and apply it to the required set of nodes. **Procedure** 1. Label a node by running the following command: ```terminal # oc label node perf-node.example.com cpumanager=true ``` 2. To enable CPU Manager for all compute nodes, edit the CR by running the following command: ```terminal # oc edit machineconfigpool worker ``` 3. Add the `custom-kubelet: cpumanager-enabled` label to `metadata.labels` section. ```yaml metadata: creationTimestamp: 2020-xx-xxx generation: 3 labels: custom-kubelet: cpumanager-enabled ``` 4. Create a `KubeletConfig`, `cpumanager-kubeletconfig.yaml`, custom resource (CR). Refer to the label created in the previous step to have the correct nodes updated with the new kubelet config. See the `machineConfigPoolSelector` section: ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: cpumanager-enabled spec: machineConfigPoolSelector: matchLabels: custom-kubelet: cpumanager-enabled kubeletConfig: cpuManagerPolicy: static cpuManagerReconcilePeriod: 5s ``` - `cpuManagerPolicy` specifies a policy: - `none`. This policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the scheduler does automatically. This is the default policy. - `static`. This policy allows containers in guaranteed pods with integer CPU requests. It also limits access to exclusive CPUs on the node. If `static`, you must use a lowercase `s`. - `cpuManagerReconcilePeriod` is optional. Specify the CPU Manager reconcile frequency. The default is `5s`. 5. Create the dynamic kubelet config by running the following command: ```terminal # oc create -f cpumanager-kubeletconfig.yaml ``` This adds the CPU Manager feature to the kubelet config and, if needed, the Machine Config Operator (MCO) reboots the node. To enable CPU Manager, a reboot is not needed. 6. Check for the merged kubelet config by running the following command: ```terminal # oc get machineconfig 99-worker-XXXXXX-XXXXX-XXXX-XXXXX-kubelet -o json | grep ownerReference -A7 ``` ```json {title="Example output"} "ownerReferences": [ { "apiVersion": "machineconfiguration.openshift.io/v1", "kind": "KubeletConfig", "name": "cpumanager-enabled", "uid": "7ed5616d-6b72-11e9-aae1-021e1ce18878" } ] ``` 7. Check the compute node for the updated `kubelet.conf` file by running the following command: ```terminal # oc debug node/perf-node.example.com sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager ``` The following is example output: ```terminal cpuManagerPolicy: static cpuManagerReconcilePeriod: 5s ``` - `cpuManagerPolicy` is defined when you create the `KubeletConfig` CR. - `cpuManagerReconcilePeriod` is defined when you create the `KubeletConfig` CR. 8. Create a project by running the following command: ```terminal $ oc new-project ``` 9. Create a pod that requests a core or multiple cores. Both limits and requests must have their CPU value set to a whole integer. That is the number of cores that will be dedicated to this pod: ```terminal # cat cpumanager-pod.yaml ``` ```yaml {title="Example output"} apiVersion: v1 kind: Pod metadata: generateName: cpumanager- spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: cpumanager image: gcr.io/google_containers/pause:3.2 resources: requests: cpu: 1 memory: "1G" limits: cpu: 1 memory: "1G" securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] nodeSelector: cpumanager: "true" ``` 10. Create the pod: ```terminal # oc create -f cpumanager-pod.yaml ``` **Verification** 1. Verify that the pod is scheduled to the node that you labeled by running the following command: ```terminal # oc describe pod cpumanager ``` ```terminal {title="Example output"} Name: cpumanager-6cqz7 Namespace: default Priority: 0 PriorityClassName: Node: perf-node.example.com/xxx.xx.xx.xxx ... Limits: cpu: 1 memory: 1G Requests: cpu: 1 memory: 1G ... QoS Class: Guaranteed Node-Selectors: cpumanager=true ``` 2. Verify that a CPU has been exclusively assigned to the pod by running the following command: ```terminal # oc describe node --selector='cpumanager=true' | grep -i cpumanager- -B2 ``` ```terminal {title="Example output"} NAMESPACE NAME CPU Requests CPU Limits Memory Requests Memory Limits Age cpuman cpumanager-mlrrz 1 (28%) 1 (28%) 1G (13%) 1G (13%) 27m ``` 3. Verify that the `cgroups` are set up correctly. Get the process ID (PID) of the `pause` process by running the following commands: ```terminal # oc debug node/perf-node.example.com ``` ```terminal sh-4.2# systemctl status | grep -B5 pause ``` > [!NOTE] > If the output returns multiple pause process entries, you must identify the correct pause process. ```terminal {title="Example output"} # ├─init.scope │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17 └─kubepods.slice ├─kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice │ ├─crio-b5437308f1a574c542bdf08563b865c0345c8f8c0b0a655612c.scope │ └─32706 /pause ``` 4. Verify that pods of quality of service (QoS) tier `Guaranteed` are placed within the `kubepods.slice` subdirectory by running the following commands: ```terminal # cd /sys/fs/cgroup/kubepods.slice/kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice/crio-b5437308f1ad1a7db0574c542bdf08563b865c0345c86e9585f8c0b0a655612c.scope ``` ```terminal # for i in `ls cpuset.cpus cgroup.procs` ; do echo -n "$i "; cat $i ; done ``` > [!NOTE] > Pods of other QoS tiers end up in child `cgroups` of the parent `kubepods`. ```terminal {title="Example output"} cpuset.cpus 1 tasks 32706 ``` 5. Check the allowed CPU list for the task by running the following command: ```terminal # grep ^Cpus_allowed_list /proc/32706/status ``` ```terminal {title="Example output"} Cpus_allowed_list: 1 ``` 6. Verify that another pod on the system cannot run on the core allocated for the `Guaranteed` pod. For example, to verify the pod in the `besteffort` QoS tier, run the following commands: ```terminal # cat /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc494a073_6b77_11e9_98c0_06bba5c387ea.slice/crio-c56982f57b75a2420947f0afc6cafe7534c5734efc34157525fa9abbf99e3849.scope/cpuset.cpus ``` ```terminal # oc describe node perf-node.example.com ``` ```terminal {title="Example output"} ... Capacity: attachable-volumes-aws-ebs: 39 cpu: 2 ephemeral-storage: 124768236Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 8162900Ki pods: 250 Allocatable: attachable-volumes-aws-ebs: 39 cpu: 1500m ephemeral-storage: 124768236Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 7548500Ki pods: 250 ------- ---- ------------ ---------- --------------- ------------- --- default cpumanager-6cqz7 1 (66%) 1 (66%) 1G (12%) 1G (12%) 29m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1440m (96%) 1 (66%) ``` This VM has two CPU cores. The `system-reserved` setting reserves 500 millicores, meaning that half of one core is subtracted from the total capacity of the node to arrive at the `Node Allocatable` amount. You can see that `Allocatable CPU` is 1500 millicores. This means you can run one of the CPU Manager pods since each will take one whole core. A whole core is equivalent to 1000 millicores. If you try to schedule a second pod, the system will accept the pod, but it will never be scheduled: ```terminal NAME READY STATUS RESTARTS AGE cpumanager-6cqz7 1/1 Running 0 33m cpumanager-7qc2t 0/1 Pending 0 11s ``` ## Topology Manager policies {#topology-manager-policies_using-cpu-manager-and-topology-manager} Topology Manager aligns `Pod` resources of all Quality of Service (QoS) classes by collecting topology hints from Hint Providers, such as CPU Manager and Device Manager, and using the collected hints to align the `Pod` resources. Topology Manager supports four allocation policies, which you assign in the `KubeletConfig` custom resource (CR) named `cpumanager-enabled`: `none` policy : This is the default policy and does not perform any topology alignment. `best-effort` policy : For each container in a pod with the `best-effort` topology management policy, kubelet tries to align all the required resources on a NUMA node according to the preferred NUMA node affinity for that container. Even if the allocation is not possible due to insufficient resources, the Topology Manager still admits the pod but the allocation is shared with other NUMA nodes. `restricted` policy : For each container in a pod with the `restricted` topology management policy, kubelet determines the theoretical minimum number of NUMA nodes that can fulfill the request. If the actual allocation requires more than the that number of NUMA nodes, the Topology Manager rejects the admission, placing the pod in a `Terminated` state. If the number of NUMA nodes can fulfill the request, the Topology Manager admits the pod and the pod starts running. `single-numa-node` policy : For each container in a pod with the `single-numa-node` topology management policy, kubelet admits the pod if all the resources required by the pod can be allocated on the same NUMA node. If a single NUMA node affinity is not possible, the Topology Manager rejects the pod from the node. This results in a pod in a `Terminated` state with a pod admission failure. ## Setting up Topology Manager {#setting_up_topology_manager_using-cpu-manager-and-topology-manager} To use Topology Manager, you must configure an allocation policy in the `KubeletConfig` custom resource (CR) named `cpumanager-enabled`. This file might exist if you have set up CPU Manager. If the file does not exist, you can create the file. **Prerequisites** - Configure the CPU Manager policy to be `static`. **Procedure** 1. To activate Topology Manager, configure the Topology Manager allocation policy in the custom resource. ```terminal $ oc edit KubeletConfig cpumanager-enabled ``` ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: cpumanager-enabled spec: machineConfigPoolSelector: matchLabels: custom-kubelet: cpumanager-enabled kubeletConfig: cpuManagerPolicy: static cpuManagerReconcilePeriod: 5s topologyManagerPolicy: single-numa-node ``` - `cpuManagerPolicy` must be `static` with a lowercase `s`. - `topologyManagerPolicy` specifies your selected Topology Manager allocation policy. In this example, the policy is `single-numa-node`. Acceptable values are: `default`, `best-effort`, `restricted`, `single-numa-node`. ## Pod interactions with Topology Manager policies {#pod-interactions-with-topology-manager_using-cpu-manager-and-topology-manager} The example `Pod` specs illustrate pod interactions with Topology Manager. The following pod runs in the `BestEffort` QoS class because no resource requests or limits are specified. ```yaml spec: containers: - name: nginx image: nginx ``` The next pod runs in the `Burstable` QoS class because requests are less than limits. ```yaml spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" requests: memory: "100Mi" ``` If the selected policy is anything other than `none`, Topology Manager would process all the pods and it enforces resource alignment only for the `Guaranteed` QoS `Pod` specification. When the Topology Manager policy is set to `none`, the relevant containers are pinned to any available CPU without considering NUMA affinity. This is the default behavior and it does not optimize for performance-sensitive workloads. Other values enable the use of topology awareness information from device plugins core resources, such as CPU and memory. The Topology Manager attempts to align the CPU, memory, and device allocations according to the topology of the node when the policy is set to other values than `none`. For more information about the available values, see *Topology Manager policies*. The following example pod runs in the `Guaranteed` QoS class because requests are equal to limits. ```yaml spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" example.com/device: "1" requests: memory: "200Mi" cpu: "2" example.com/device: "1" ``` Topology Manager would consider this pod. The Topology Manager would consult the Hint Providers, which are the CPU Manager, the Device Manager, and the Memory Manager, to get topology hints for the pod. Topology Manager will use this information to store the best topology for this container. In the case of this pod, CPU Manager and Device Manager will use this stored information at the resource allocation stage. **Additional resources** {._additional-resources} - [Topology Manager policies](/openshift-docs-markdown/scalability_and_performance/using-cpu-manager#topology-manager-policies_using-cpu-manager-and-topology-manager)