Integrating Dynamic Resource Allocation

You can configure Red Hat build of Kueue to manage quota for workloads that use Dynamic Resource Allocation (DRA) to request GPUs. When DRA quota management is configured, Red Hat build of Kueue counts DRA device requests toward quota in the same way that it counts traditional resources such as CPU and memory.

warning

Kueue integration with Dynamic Resource Allocation (DRA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If DRA device quota is not configured, Red Hat build of Kueue does not account for GPU requests when admitting workloads, which can result in teams exceeding their GPU allocation.

DRA quota management overview

Dynamic Resource Allocation (DRA) is a Kubernetes framework that provides structured discovery and allocation of specialized hardware such as GPUs. DRA drivers publish device information through ResourceSlice objects, and administrators group devices into categories using DeviceClass objects.

Without Red Hat build of Kueue DRA integration, GPU requests made through DRA are invisible to quota management. Red Hat build of Kueue cannot account for these requests when admitting workloads, which can result in teams exceeding their GPU allocation.

Red Hat build of Kueue provides two approaches for managing DRA device quota:

ResourceClaimTemplate: The default approach. Workloads explicitly reference a ResourceClaimTemplate object that defines device requirements. Administrators configure deviceClassMappings in the Kueue CR to map each DeviceClass object to a logical resource name for quota tracking. Use this approach when workloads need fine-grained control over device selection, such as targeting a specific GPU model or architecture using CEL selectors.
Extended resources: A simplified alternative that allows workloads to use standard Kubernetes resources.requests syntax, for example, nvidia.com/gpu: "1", instead of explicitly creating DRA objects. When a DeviceClass object includes the spec.extendedResourceName field, the Kubernetes scheduler automatically generates ResourceClaim objects. Use this approach when you want the simplest possible user experience and backward compatibility with existing workload YAML.
note
If a DeviceClass object with the extendedResourceName field also appears in a deviceClassMappings entry, Red Hat build of Kueue uses the mapped logical name from the deviceClassMappings entry for quota instead of the extended resource name, unifying quota accounting across both paths.

For clusters with partitionable devices, such as NVIDIA Multi-Instance GPU (MIG), Red Hat build of Kueue can also charge quota in capacity units, such as GPU memory, rather than device count. Partitionable devices use ResourceClaimTemplates objects with CEL selectors to target specific partition profiles, and require administrators to configure counter-based sources in deviceClassMappings. This capability requires OpenShift Container Platform 4.22 or later.

Configuring the resource claim template path

You can configure Red Hat build of Kueue to manage quota for workloads that explicitly reference ResourceClaimTemplate objects. This requires configuring the deviceClassMappings entry in the Red Hat build of Kueue custom resource (CR) and adding the DRA resource to your ClusterQueue object.

Prerequisites

You have installed Red Hat build of Kueue by using the Red Hat Build of Kueue Operator.
You have created a Kueue custom resource (CR).
Your cluster is running OpenShift Container Platform 4.21 or later.
A DRA driver is installed in the cluster, for example, nvidia-dra-driver. You can verify that the DRA driver is publishing device information by running the following command:
```
$ oc get resourceslices
```
If the command returns one or more ResourceSlice objects, the DRA driver is running.
At least one DeviceClass object exists in the cluster. You can verify this by running the following command:
```
$ oc get deviceclass
```

Procedure

Use the following command to add a deviceClassMappings entry to the Red Hat build of Kueue configuration that maps each DeviceClass to a logical resource name for quota:
```
$ oc patch kueue cluster -n openshift-kueue-operator --type=merge -p '{
  "spec": {
    "config": {
      "resources": {
        "deviceClassMappings": [{
          "name": "nvidia.com/gpu",
          "deviceClassNames": ["gpu.nvidia.com"]
        }]
      }
    }
  }
}'
```
Replace "nvidia.com/gpu" with the resource name used in ClusterQueue quotas and Workload status.

Replace "gpu.nvidia.com" with one or more DeviceClass names that map to this resource.

Multiple device classes can map to the same logical resource name. For example, if you have separate device classes for different GPU models but want a single quota pool, as shown in the following example:
```
resources:
  deviceClassMappings:
  - name: nvidia.com/gpu
    deviceClassNames:
    - gpu-a100.nvidia.com
    - gpu-h100.nvidia.com
```

Create a file called rct-queues.yaml that contains the following content:

Example quota configuration for a ResourceClaimTemplate object
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 40
      - name: "memory"
        nominalQuota: 200Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 8
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"

Apply the rct-queues.yaml file:
```
$ oc apply -f rct-queues.yaml
```

Create a ResourceClaimTemplate object and a workload to verify the configuration. Create a file called rct-job.yaml by running the following command:

$ oc create -f rct-job.yaml

Example ResourceClaimTemplate workload
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: my-gpu
  namespace: default
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: rct-test-job-
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
      - name: gpu
        resourceClaimTemplateName: my-gpu
      containers:
      - name: worker
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          claims:
          - name: gpu
          requests:
            cpu: "1"
            memory: "200Mi"

where:

spec.spec.drivers.requests.exactly.deviceClassName:: References the DeviceClass object configured in the deviceClassMappings entry.
metadata.labels.kueue.x-k8s.io/queue-name:: Identifies the local queue to submit the job to.
spec.template.spec.resourceClaims.resourceClaimTemplateName:: References the ResourceClaimTemplate object defined above. The template must exist in the same namespace as the job.
spec.template.spec.containers.resources.claims.name:: Attaches the resource claim to this container.

Verification

Verify that the workload has been created and admitted:
```
$ oc -n default get workloads
```
Verify that a ResourceClaim object was created from the template:
```
$ oc -n default get resourceclaims
```
If the workload is not admitted, verify the following:
- Check if the namespace is managed by Red Hat build of Kueue:
```
$ oc label namespace default kueue.openshift.io/managed=true
```
- The deviceClassMappings in the Kueue CR maps the DeviceClass object to the resource name in the coveredResources parameter.
- The ClusterQueue object has sufficient quota available.
- The ResourceClaimTemplate object exists in the same namespace as the job.

Configuring the extended resources path

You can configure Red Hat build of Kueue to manage quota for workloads that request GPUs by using the standard resources.requests syntax, for example, nvidia.com/gpu: "1".

When a DeviceClass object includes the spec.extendedResourceName field, the Kubernetes scheduler automatically generates ResourceClaim objects. This path does not require deviceClassMappings configuration because Red Hat build of Kueue auto-discovers the mapping by indexing DeviceClass objects.

note

The Red Hat build of Kueue Operator automatically enables the required Red Hat build of Kueue feature gates when it detects the DRAExtendedResource Kubernetes feature gate on the cluster. No manual Red Hat build of Kueue feature gate configuration is required.

To use the extended resources path, you must enable the DRAExtendedResource Kubernetes feature gate. This feature is expected to be generally available in a future OpenShift Container Platform release.

Prerequisites

You have cluster administrator permissions.
You have installed Red Hat build of Kueue by using the Red Hat Build of Kueue Operator.
You have created a Kueue CR.
Your cluster is running OpenShift Container Platform 4.21 or later.
A DRA driver is installed in the cluster, for example, nvidia-dra-driver. You can verify that the DRA driver is publishing device information by running the following command:
```
$ oc get resourceslices
```
If the command returns one or more ResourceSlice objects, the DRA driver is running.
At least one DeviceClass object exists in the cluster. You can verify this by running the following command:
```
$ oc get deviceclass
```
You have enabled the DRAExtendedResource Kubernetes feature gate by adding the CustomNoUpgrade feature set to the FeatureGate CR named cluster, as shown in the following example:
```
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster
spec:
  featureSet: CustomNoUpgrade
  customNoUpgrade:
    enabled:
    - DRAExtendedResource
```
warning
Enabling the CustomNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. This feature set is not supported on production clusters.

Procedure

Verify that the DeviceClass object has spec.extendedResourceName set by running the following command:
```
$ oc get deviceclass gpu.nvidia.com -o jsonpath='{.spec.extendedResourceName}'
```
Example output
```
nvidia.com/gpu
```
If the command does not return a value, add the extendedResourceName field by running the following command:
```
$ oc patch deviceclass gpu.nvidia.com --type=merge -p '{"spec":{"extendedResourceName":"nvidia.com/gpu"}}'
```

Create a ClusterQueue object that includes the GPU resource in the coveredResources parameter by creating a file called er-queues.yaml, as shown in the following example:

Example quota configuration for extended resources
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 40
      - name: "memory"
        nominalQuota: 200Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 8
---
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    kueue.openshift.io/managed: "true"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "team-a"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"

Apply the quota configuration by running the following command:
```
$ oc apply -f er-queues.yaml
```

Create a workload that uses the standard resource request syntax by creating a file called er-job.yaml, as shown in the following example:

Example workload using extended resources
apiVersion: batch/v1
kind: Job
metadata:
  name: er-test-job
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  template:
    spec:
      containers:
      - name: worker
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
            nvidia.com/gpu: "1"
          limits:
            nvidia.com/gpu: "1"
      restartPolicy: Never

where:

metadata.labels.kueue.x-k8s.io/queue-name: Identifies the local queue to submit the job to.

spec.template.spec.containers.resources.requests.cpu.nvidia.com/gpu: Requests a GPU by using the standard extended resource syntax. No ResourceClaimTemplate or resourceClaims section is needed. The DeviceClass object with the spec.extendedResourceName field causes the Kubernetes scheduler to generate a ResourceClaim object automatically.
spec.template.spec.containers.resources.limits.cpu.nvidia.com/gpu: Replace "1" with a GPU by using the standard extended resource syntax. No ResourceClaimTemplate or resourceClaims section is needed. The DeviceClass object with the spec.extendedResourceName field causes the Kubernetes scheduler to generate a ResourceClaim object automatically.

Create the workload by running the following command:
```
$ oc apply -f er-job.yaml
```

Verification

Verify that a workload has been created and admitted by running the following command:

$ oc -n team-a get workloads

Example output
NAME                          QUEUE        RESERVED IN     ADMITTED   AGE
job-er-test-job-4m2x-d3f4g   user-queue   cluster-queue   True       10s

Verify that a ResourceClaim was automatically created by running the following command:
```
$ oc -n team-a get resourceclaims
```
Example output
```
NAME                                         STATE                AGE
er-test-job-jj7vz-extended-resources-bggzk   allocated,reserved   24s
```
The Kubernetes scheduler creates a ResourceClaim for each pod that requests an extended resource backed by a DeviceClass.

If the workload is not admitted, verify the following:
- The DRAExtendedResource Kubernetes feature gate is enabled on the cluster.
- The DeviceClass has spec.extendedResourceName set.
- The ClusterQueue includes the extended resource name in coveredResources.
- The ClusterQueue has sufficient quota available.

Configuring the partitionable devices

You can configure Red Hat build of Kueue to manage quota for partitionable devices based on actual device capacity rather than device count. Partitionable devices, such as NVIDIA Multi-Instance GPU (MIG) capable GPUs, allow a single GPU to be dynamically subdivided into smaller partitions.

When counter-based quota is configured, Red Hat build of Kueue charges quota in capacity units such as GPU memory rather than counting whole devices. For example, a 1g.5gb MIG partition on an A100-40GB charges 4864Mi of GPU memory quota, while a whole GPU charges 40320Mi.

note

To use partitionable devices, your cluster must be running OpenShift Container Platform 4.22 or later and must have the CustomNoUpgrade feature set enabled with explicit DRAPartitionableDevices gate enablement.

Prerequisites

You have cluster administrator permissions.
You have installed Red Hat build of Kueue by using the Red Hat Build of Kueue Operator.
You have created a Kueue custom resource (CR).
Your cluster is running OpenShift Container Platform 4.22 or later.
A DRA driver that publishes consumesCounters in ResourceSlice objects is installed, for example, nvidia-dra-driver. You can verify that the DRA driver is publishing device information by running the following command:
```
$ oc get resourceslices
```
If the command returns one or more ResourceSlice objects, the DRA driver is running.
At least one DeviceClass object exists in the cluster. You can verify this by running the following command:
```
$ oc get deviceclass
```
MIG is enabled on the GPU hardware.
You have enabled the DRAPartitionableDevices Kubernetes feature gate by adding the CustomNoUpgrade feature set to the FeatureGate CR named cluster, as shown in the following example:
```
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster
spec:
  featureSet: CustomNoUpgrade
  customNoUpgrade:
    enabled:
    - DRAPartitionableDevices
```
warning
Enabling the CustomNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. This feature set is not supported on production clusters. For information about enabling feature gates, see "Enabling features using feature gates".

Procedure

Verify that your DRA driver publishes counter data by running the following command:
```
$ oc get resourceslices -o jsonpath='{range .items[*]}{.spec.driver}{"\t"}{range .spec.devices[*]}{.name}: {.consumesCounters}{"\n"}{end}{end}'
```
Example output
```
gpu.nvidia.com	gpu-0: [{"counterSet":"shared","counters":{"memory":{"value":"40Gi"}}}]
```
If the output does not show consumesCounters data, verify that your DRA driver version supports partitionable devices and that MIG is enabled on the GPU hardware.

Configure counter-based quota by adding a deviceClassMappings entry with a sources section to the config.resources section of the Red Hat build of Kueue CR, as shown in the following example:

apiVersion: kueue.openshift.io/v1
kind: Kueue
metadata:
  name: cluster
  namespace: openshift-kueue-operator
spec:
  config:
    resources:
      deviceClassMappings:
      - name: gpu.memory
        deviceClassNames:
        - gpu.nvidia.com
        - mig.nvidia.com
        sources:
        - type: Counter
          counter:
            name: memory
            driver: gpu.nvidia.com
            deviceSelector:
              type: CEL
              cel:
                expression: "device.driver == 'gpu.nvidia.com'"
# ...

where:

spec.config.resources.deviceClassMappings.name: The logical resource name used in ClusterQueue quotas. When counter-based sources are configured, quota is charged in capacity units rather than device count.

spec.config.resources.deviceClassMappings.deviceClassNames: The DeviceClass names that map to this resource. Include both the whole-GPU class (gpu.nvidia.com) and the MIG class (mig.nvidia.com).
spec.config.resources.deviceClassMappings.sources: Defines how Red Hat build of Kueue computes the quota charge.
spec.config.resources.deviceClassMappings.sources.counter.name: The counter name must match a counter key published by the DRA driver in ResourceSlice devices.
spec.config.resources.deviceClassMappings.sources.counter.deviceSelector: Scopes which devices are eligible for counter-based quota accounting.
note
The Red Hat build of Kueue Operator automatically enables the required Red Hat build of Kueue feature gates when it detects the DRAPartitionableDevices Kubernetes feature gate and sources are configured in deviceClassMappings. No manual Red Hat build of Kueue feature gate configuration is required.

Create a ClusterQueue object with counter-based quota. Set the quota in capacity units rather than device count. Create a file called pd-queues.yaml with the following content:

Example quota configuration for partitionable devices
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "gpu.memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 40
      - name: "memory"
        nominalQuota: 200Gi
      - name: "gpu.memory"
        nominalQuota: 800Gi
---
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    kueue.openshift.io/managed: "true"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "team-a"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"

where:

spec.resourceGroups.coveredResources: The gpu.memory entry must match the name value in deviceClassMappings.

spec.resourceGroups.flavors.resources.name: Specify "gpu.memory" to set the total GPU memory quota. For example, 800Gi accommodates twenty A100-40GB GPUs or equivalent MIG partitions.
note
When ClusterQueue objects share a cohort, ensure all queues use the same unit scale for counter resources. Red Hat build of Kueue does not validate unit consistency across ClusterQueue objects.

Apply the quota configuration by running the following command:
```
$ oc apply -f pd-queues.yaml
```

Create a workload that requests a MIG partition by creating a file called pd-job.yaml, as shown in the following example:

Example workload requesting a MIG partition
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: team-a
  name: gpu-partition
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: mig.nvidia.com
          count: 1
          selectors:
          - cel:
              expression: "device.attributes['gpu.nvidia.com'].profile == '1g.5gb'"
---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: pd-test-job
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  template:
    spec:
      containers:
      - name: worker
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          claims:
          - name: gpu
          requests:
            cpu: "1"
            memory: "200Mi"
      resourceClaims:
      - name: gpu
        resourceClaimTemplateName: gpu-partition
      restartPolicy: Never

where:

spec.spec.devices.requests.exactly.deviceClassName: References the MIG DeviceClass.

spec.spec.devices.requests.exactly.selectors.cel.expression:: Selects a specific MIG partition profile. Available profiles depend on the GPU model, for example, 1g.5gb, 2g.10gb, 3g.20gb, or 7g.40gb for the A100-40GB.
metadata.labels.kueue.x-k8s.io/queue-name:: Identifies the local queue to submit the job to.
spec.template.spec.resourceClaims.resourceClaimTemplateName:: References the ResourceClaimTemplate defined above. The ResourceClaimTemplate must exist in the same namespace as the job.

Create the workload by running the following command:
```
$ oc create -f pd-job.yaml
```

Verification

Verify that the workload is admitted and that quota was charged in capacity units by running the following command:
```
$ oc -n team-a get workloads -o jsonpath='{range .items[*]}{.metadata.name}: {.status.admission.podSetAssignments[0].resourceUsage}{"\n"}{end}'
```
Example output
```
job-pd-test-job-xxxxx: {"cpu":"1","gpu.memory":"5100273664","memory":"200Mi"}
```
The gpu.memory value reflects the actual memory capacity of the requested MIG partition rather than a device count of 1.
If the workload is not admitted, verify the following:
- The DRAPartitionableDevices Kubernetes feature gate is enabled on the cluster.
- The name value of the deviceClassMappings object matches the resource name in coveredResources.
- The counter.name in sources matches a counter key in the ResourceSlice objects.
- The ClusterQueue has sufficient GPU memory quota for the requested partition size.
- MIG is enabled on the GPU hardware.

Additional resources

DRA quota management overview​

Configuring the resource claim template path​

Configuring the extended resources path​

Configuring the partitionable devices​

DRA quota management overview

Configuring the resource claim template path

Configuring the extended resources path

Configuring the partitionable devices