Skip to content

Release notes

Red Hat build of Kueue is released as an Operator that is supported on OpenShift Container Platform.

Compatible environments

Your cluster must meet specific architecture and platform requirements before you can install Red Hat build of Kueue.

Supported architectures

Red Hat build of Kueue version 1.1 and later is supported on the following architectures:

  • ARM64
  • 64-bit x86
  • ppc64le (IBM Power(R))
  • s390x (IBM Z(R))

Supported platforms

Red Hat build of Kueue version 1.1 and later is supported on the following platforms:

  • OpenShift Container Platform
  • Hosted control planes for OpenShift Container Platform

Warning

Currently, Red Hat build of Kueue is not supported on Red Hat build of MicroShift (MicroShift).

Release notes for Red Hat build of Kueue version 1.4.2

Red Hat build of Kueue version 1.4.2 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.4.2 uses Kueue version 0.18.

Fixed issues

Updates Operator and Operand images to the latest Red Hat Universal Base Image (UBI) layers to incorporate security fixes.

Release notes for Red Hat build of Kueue version 1.3.2

Red Hat build of Kueue version 1.3.2 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.3 uses Kueue version 0.16.

Fixed issues

Updates Operator and Operand images to the latest Red Hat Universal Base Image (UBI) layers to incorporate security fixes.

Release notes for Red Hat build of Kueue version 1.4.1

Red Hat build of Kueue version 1.4.1 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.4.1 uses Kueue version 0.18.

Fixed issues

Red Hat build of Kueue no longer accepts invalid webhook configurations at admission time

Before this update, Red Hat build of Kueue filtered out core validating webhooks during reconciliation. As a consequence, the Operator silently accepted invalid webhook configurations for the following resources:

  • Cohort
  • ClusterQueue
  • Workload
  • ResourceFlavor

With this release, the Operator always registers validating webhooks. As a result, Red Hat build of Kueue rejects invalid configurations at admission time.

(OCPBUGS-99316)

Release notes for Red Hat build of Kueue version 1.4

Red Hat build of Kueue version 1.4 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.4 uses Kueue version 0.18.

New features and enhancements

Admission fair sharing

This release introduces admission fair sharing, which balances workload admission across multiple local Queues feeding into a shared ClusterQueue. Admission fair sharing:

  • Prioritizes workloads based on historical resource consumption
  • Tracks usage over time with a configurable decay function
  • Applies immediate admission penalties to prevent resource monopolization

For more information, see Admission fair sharing.

Dynamic Resource Allocation (DRA) quota management for GPUs (Technology Preview)
You can manage quotas for workloads that request GPUs through Dynamic Resource Allocation (DRA). When quota management is configured, Red Hat build of Kueue tracks DRA device requests toward quota alongside traditional resources such as CPU and memory, preventing teams from exceeding their allocated GPU resources.

For more information, see Integrating Dynamic Resource Allocation.

Fixed issues

Use the resourceNames object to limit webhooks to only Red Hat build of Kueue resources

You can restrict the kueue-manager-role ClusterRole webhook configurations and CRD rules to specific resourceNames, preventing the controller from modifying other Operators' webhook configurations or CRDs. Webhook rules are scoped to kueue-mutating-webhook-configuration and kueue-validating-webhook-configuration, as shown in this example:

resourceNames: 
  - kueue-mutating-webhook-configuration
  - kueue-validating-webhook-configuration

(OCPBUGS-88495)

Removed secrets from the core API resources list

The upstream version of Kueue moved the secrets RBAC to a namespace-scoped role (kueue-manager-secrets-role), but the ClusterRole was not updated to remove the cluster-wide secrets permission.

This version of Red Hat build of Kueue removes the secrets resource type from the cluster-wide openshift-kueue-operator ClusterRole. The namespace-scoped kueue-manager-secrets-role role already exists and provides the necessary access.

(OCPBUGS-88040)

Release notes for Red Hat build of Kueue version 1.3.1

Red Hat build of Kueue version 1.3.1 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.3 uses Kueue version 0.16.5.

Fixed issues

kueue.x-k8s.io/queue-name refers to a non-existent queue

Fixed a bug where referencing a non-existent LocalQueue via kueue.x-k8s.io/queue-name could cause a running pod to be terminated and permanently stuck with unremovable scheduling gates.

(OCPBUGS-78789)

Release notes for Red Hat build of Kueue version 1.3

Red Hat build of Kueue version 1.3 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.3 uses Kueue version 0.16.

New features and enhancements

Leader Worker Set Operator
Red Hat build of Kueue version 1.3 provides for the integration of the Leader Worker Set Operator with Red Hat build of Kueue so you can leverage the Red Hat build of Kueue scheduling and resource management functionality when running LeaderWorkerSets. For more information, see Integrating the Leader Worker Set Operator.
JobSet Operator
Red Hat build of Kueue version 1.3 provides for the integration of the JobSet Operator so you can use the JobSet Operator to manage and run large-scale, coordinated workloads like high-performance computing (HPC) and AI training. For more information, see Integrating the JobSet Operator.
Upstream progression of the Red Hat build of Kueue API to v1beta2

Red Hat build of Kueue version 1.3 provides the v1beta2 version of the Red Hat build of Kueue API. This update continues the evolution of the Red Hat build of Kueue APIs with the ultimate goal of graduating the API to v1.

All new Kueue objects created after the upgrade will be stored using the v1beta2 version. The earlier version of the API, v1beta1 is deprecated. Objects can still be created using v1beta1, if necessary. In these cases, a deprecation message is shown.

However, existing objects are only auto-converted to the new storage version by Kubernetes during a write request. This means that Red Hat build of Kueue API objects that rarely receive updates such as Topologies, ResourceFlavors, or long-running Workloads could remain in the older v1beta1 format indefinitely.

Fixed issues

Reconcile jobs only in opt-in namespaces

OpenShift Container Platform allowed reconciliation of Job resources that have the kueue.x-k8s.io/queue-name label, even if these resources are in namespaces that are not configured to opt in to being managed by OpenShift Container Platform. With this release, there is ongoing upstream work that updates this behavior so that Jobs with queue-name labels are also ignored unless their namespace matches the managedJobsNamespaceSelector. This change makes Red Hat build of Kueue behavior consistent across all integrations.

(OCPBUGS-58205)

Kueue CR description reads as "Not available" in the OpenShift Container Platform web console

After installing Red Hat build of Kueue, in the Operator details view, the description for the Kueue CR read as "Not available". This issue did not affect or degrade the Red Hat build of Kueue Operator functionality. With this release, the "Not available" message no longer displays.

(OCPBUGS-62185)

LeaderWorkerSet and Jobset validation errors

Currently, the Leader Worker Set Operator and JobSet Operator are only validated after the Operand CR is updated and the full Kueue hierarchy (ResourceFlavor, ClusterQueue, and LocalQueue) is established. Any configuration errors appear only when applying a LeaderWorkerSet or JobSet template.

(OCPBUGS-74210)

Known issues

LeaderWorkerSet pods update sequentially by default

If you have integrated Leader Worker Set Operator with your Red Hat build of Kueue installation and you are using the rollout strategy option for updating LeaderWorkerSet pods, be aware that the MaxUnavailable feature gate in OpenShift Container Platform is disabled by default.

When any change is made to LeaderWorkerSet pods, a rolling update is triggered. This action gradually replaces the old pods of a deployment with new ones, keeping as many pods alive as possible to avoid downtime. If MaxUnavailable is disabled, which is the OpenShift Container Platform default setting, the pods are updated one at a time.

If you want to run updates in parallel instead of running them sequentially, MaxUnavailable feature gate must be enabled. For more information, see Enabling feature sets at installation and Rollout Strategy.

Release notes for Red Hat build of Kueue version 1.2

Red Hat build of Kueue version 1.2 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.2 uses Kueue version 0.14.

New features and enhancements

Monitoring of pending workloads
Red Hat build of Kueue version 1.2 provides the VisibilityOnDemand feature to monitor the pipeline of pending jobs in the cluster queue and the local queue, and help users to estimate when their jobs will start. For more information, see Monitoring pending workloads.

Fixed issues

Custom resources are not deleted properly when you uninstall Red Hat build of Kueue

After you uninstall the Red Hat Build of Kueue Operator using the Delete all operand instances for this operator option in the OpenShift Container Platform web console, Red Hat build of Kueue custom resources were attempted to be deleted. With this release, they are not considered for deletion.

(OCPBUGS-62254)

Documentation error in previous versions of Red Hat build of Kueue

In Creating a Kueue custom resource, the optional workload types Pod, Deployment, StatefulSet were omitted. They are now included.

(OCPBUGS-62877)

Red Hat build of Kueue metrics were not being exposed to Prometheus from version 1.1

Prometheus was not scraping metrics from the Operator’s controller, even though the ServiceMonitor and RBAC resources were successfully created as part of the Operator installation. As a result, none of the Kueue metrics were available in the cluster monitoring stack.

The metrics service added during the installation was configured with an incorrect port reference, causing Prometheus to fail in scraping metrics from the Kueue endpoint. The port name has been updated with the correct port name.

(OCPBUGS-63441)

Known issues

Reconcile jobs only in opt-in namespaces

OpenShift Container Platform allows reconciliation of Job resources that have the kueue.x-k8s.io/queue-name label, even if these resources are in namespaces which are not configured to opt in to being managed by OpenShift Container Platform. This is inconsistent with the behavior for other core integrations like pods, deployments, and stateful sets, which are only reconciled if they are in namespaces which have been configured to opt in to being managed by OpenShift Container Platform by adding the kueue.openshift.io/managed=true.

(OCPBUGS-58205)

Kueue CR description reads as "Not available" in the OpenShift Container Platform web console

After installing Red Hat build of Kueue, in the Operator details view, the description for the Kueue CR reads as "Not available". This issue does not affect or degrade the Red Hat build of Kueue Operator functionality.

(OCPBUGS-62185)

Release notes for Red Hat build of Kueue version 1.1

Red Hat build of Kueue version 1.1 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.1 uses Kueue version 0.12.

Warning

If you have a previously installed version of Red Hat build of Kueue on your cluster, you must uninstall the Operator and manually install version 1.1. For information see Upgrading Red Hat build of Kueue.

New features and enhancements

Configure a default local queue

A default local queue serves as the local queue for newly created jobs that do not have the kueue.x-k8s.io/queue-name label. After you create a default local queue, any new jobs created in the namespace without a kueue.x-k8s.io/queue-name label automatically update to have the kueue.x-k8s.io/queue-name: default label.

(RFE-7615)

Multi-architecture and Hosted control planes support

With this release, Red Hat build of Kueue is supported on multiple different architectures, including ARM64, 64-bit x86, ppc64le (IBM Power(R)), and s390x (IBM Z(R)), as well as on Hosted control planes for OpenShift Container Platform.

(OCPSTRAT-2103)

(OCPSTRAT-2106)

Fixed issues

You can create a Kueue custom resource by using the OpenShift Container Platform web console

Before this update, if you tried to use the OpenShift Container Platform web console to create a Kueue custom resource (CR) by using the form view, the web console showed an error and the resource could not be created. With this release, the default namespace was removed from the Kueue CR template. As a result, you can use the OpenShift Container Platform web console to create a Kueue CR by using the form view.

(OCPBUGS-58118)

Known issues

Kueue CR description reads as "Not available" in the OpenShift Container Platform web console

After you install Red Hat build of Kueue, in the Operator details view, the description for the Kueue CR reads as "Not available". This issue does not affect or degrade the Red Hat build of Kueue Operator functionality.

(OCPBUGS-62185)

Custom resources are not deleted properly when you uninstall Red Hat build of Kueue

After you uninstall the Red Hat Build of Kueue Operator using the Delete all operand instances for this operator option in the OpenShift Container Platform web console, some Red Hat build of Kueue custom resources are not fully deleted. These resources can be viewed in the Installed Operators view with the status Resource is being deleted. As a workaround, you can manually delete the resource finalizers to remove them fully.

(OCPBUGS-62254)

Release notes for Red Hat build of Kueue version 1.0.1

Red Hat build of Kueue version 1.0.1 is a patch release that is supported on OpenShift Container Platform versions 4.18 and 4.19 on the 64-bit x86 architecture.

Red Hat build of Kueue version 1.0.1 uses Kueue version 0.11.

Bug fixes in Red Hat build of Kueue version 1.0.1

  • Previously, leader election for Red Hat build of Kueue was not configured to tolerate disruption, which resulted in frequent crashing. With this release, the leader election values for Red Hat build of Kueue have been updated to match the durations recommended for OpenShift Container Platform. (OCPBUGS-58496)
  • Previously, the ReadyReplicas count was not set in the reconciler, which meant that the Red Hat build of Kueue Operator status would report that there were no replicas ready. With this release, the ReadyReplicas count is based on the number of ready replicas for the deployment, which ensures that the Operator shows as ready in the OpenShift Container Platform console when the kueue-controller-manager pods are ready. (OCPBUGS-59261)
  • Previously, when the Kueue custom resource (CR) was deleted from the openshift-kueue-operator namespace, the kueue-manager-config config map was not deleted automatically and could remain in the namespace. With this release, the kueue-manager-config config map, kueue-webhook-server-cert secret, and metrics-server-cert secret are deleted automatically when the Kueue CR is deleted. (OCPBUGS-57960)

Release notes for Red Hat build of Kueue version 1.0

Red Hat build of Kueue version 1.0 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and 4.19 on the 64-bit x86 architecture. Red Hat build of Kueue version 1.0 uses Kueue version 0.11.

New features and enhancements

Role-based access control (RBAC)
Role-based access control (RBAC) enables you to control which types of users can create which types of Red Hat build of Kueue resources.
Configure resource quotas
Configuring resource quotas by creating cluster queues, resource flavors, and local queues enables you to control the amount of resources used by user-submitted jobs and workloads.
Control job and workload management
Labeling namespaces and configuring label policies enable you to control which jobs and workloads are managed by Red Hat build of Kueue.
Share borrowable resources between queues
Configuring cohorts, fair sharing, and gang scheduling settings enable you to share unused, borrowable resources between queues.

Known issues

Jobs in all namespaces are reconciled if they have the kueue.x-k8s.io/queue-name label

Red Hat build of Kueue uses the managedJobsNamespaceSelector configuration field, so that administrators can configure which namespaces opt in to be managed by Red Hat build of Kueue. Because namespaces must be manually configured to opt in to being managed by Red Hat build of Kueue, resources in system or third-party namespaces are not impacted or managed by Red Hat build of Kueue.

The behavior in Red Hat build of Kueue 1.0 allows reconciliation of Job resources that have the kueue.x-k8s.io/queue-name label, even if these resources are in namespaces that are not configured to opt in to being managed by Red Hat build of Kueue. This is inconsistent with the behavior for other core integrations like pods, deployments, and stateful sets, which are only reconciled if they are in namespaces that have been configured to opt in to being managed by Red Hat build of Kueue.

(OCPBUGS-58205)

You cannot create a Kueue custom resource by using the OpenShift Container Platform web console

If you try to use the OpenShift Container Platform web console to create a Kueue custom resource (CR) by using the form view, the web console shows an error and the resource cannot be created. As a workaround, use the YAML view to create a Kueue CR instead.

(OCPBUGS-58118)

Additional resources