---
title: Release notes
---

# Release notes {#release-notes}

Red Hat build of Kueue is released as an Operator that is supported on OpenShift Container Platform.

## Compatible environments {#compatible-environments_release-notes}

Your cluster must meet specific architecture and platform requirements before you can install Red Hat build of Kueue.

### Supported architectures {#compatible-environments-arch_release-notes}

Red Hat build of Kueue version 1.1 and later is supported on the following architectures:

- ARM64
- 64-bit x86
- ppc64le (IBM Power(R))
- s390x (IBM Z(R))

### Supported platforms {#compatible-environments-platforms_release-notes}

Red Hat build of Kueue version 1.1 and later is supported on the following platforms:

- OpenShift Container Platform
- Hosted control planes for OpenShift Container Platform

> [!IMPORTANT]
> Currently, Red Hat build of Kueue is not supported on Red Hat build of MicroShift (MicroShift).

## Release notes for Red Hat build of Kueue version 1.3.2 {#release-notes-1.3.2_release-notes}

Red Hat build of Kueue version 1.3.2 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.3 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.16.

### Fixed issues {#release-notes-1.3.2-fixed-issues_release-notes}

Updates Operator and Operand images to the latest Red Hat Universal Base Image (UBI) layers to incorporate security fixes.

## Release notes for Red Hat build of Kueue version 1.4.1 {#release-notes-1.4.1_release-notes}

Red Hat build of Kueue version 1.4.1 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.4.1 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.18.

### Fixed issues {#release-notes-1.4.1-fixed-issues_release-notes}

Red Hat build of Kueue no longer accepts invalid webhook configurations at admission time
:   Before this update, Red Hat build of Kueue filtered out core validating webhooks during reconciliation. As a consequence, the Operator silently accepted invalid webhook configurations for the following resources:

    - `Cohort`
    - `ClusterQueue`
    - `Workload`
    - `ResourceFlavor`

With this release, the Operator always registers validating webhooks. As a result, Red Hat build of Kueue rejects invalid configurations at admission time.

([OCPBUGS-99316](https://redhat.atlassian.net/browse/OCPBUGS-99316))

## Release notes for Red Hat build of Kueue version 1.4 {#release-notes-1.4_release-notes}

Red Hat build of Kueue version 1.4 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.4 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.18.

### New features and enhancements {#release-notes-1.4-new-features_release-notes}

Admission fair sharing
:   This release introduces admission fair sharing, which balances workload admission across multiple local Queues feeding into a shared `ClusterQueue`. Admission fair sharing:

    - Prioritizes workloads based on historical resource consumption
    - Tracks usage over time with a configurable decay function
    - Applies immediate admission penalties to prevent resource monopolization

For more information, see [Admission fair sharing](/openshift-docs-markdown/ai_workloads/kueue/admission-fair-sharing#admission-fair-sharing).

Dynamic Resource Allocation (DRA) quota management for GPUs (Technology Preview)
:   You can manage quotas for workloads that request GPUs through Dynamic Resource Allocation (DRA). When quota management is configured, Red Hat build of Kueue tracks DRA device requests toward quota alongside traditional resources such as CPU and memory, preventing teams from exceeding their allocated GPU resources.

For more information, see [Integrating Dynamic Resource Allocation](/openshift-docs-markdown/ai_workloads/kueue/kueue-dra-integrating-dynamic-resource-allocation#kueue-dra-integrating-dynamic-resource-allocation).

### Fixed issues {#release-notes-1.4-fixed-issues_release-notes}

Use  the `resourceNames` object to limit webhooks to only Red Hat build of Kueue resources
:   You can restrict the `kueue-manager-role` `ClusterRole` webhook configurations and CRD rules to specific `resourceNames`, preventing the controller from modifying other Operators' webhook configurations or CRDs. Webhook rules are scoped to `kueue-mutating-webhook-configuration` and `kueue-validating-webhook-configuration`, as shown in this example:

    ```yaml
    resourceNames:
      - kueue-mutating-webhook-configuration
      - kueue-validating-webhook-configuration
    ```

    ([OCPBUGS-88495](https://issues.redhat.com/browse/OCPBUGS-88495))

Removed secrets from the core API resources list
:   The upstream version of Kueue moved the `secrets` RBAC to a namespace-scoped role (`kueue-manager-secrets-role`), but the `ClusterRole` was not updated to remove the cluster-wide secrets permission.

    This version of Red Hat build of Kueue removes the `secrets` resource type from the cluster-wide openshift-kueue-operator `ClusterRole`. The namespace-scoped `kueue-manager-secrets-role` role already exists and provides the necessary access.

    ([OCPBUGS-88040](https://issues.redhat.com/browse/OCPBUGS-88040))

## Release notes for Red Hat build of Kueue version 1.3.1 {#release-notes-1.3.1_release-notes}

Red Hat build of Kueue version 1.3.1 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.3 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.16.5.

### Fixed issues {#release-notes-1.3.1-fixed-issues_release-notes}

kueue.x-k8s.io/queue-name refers to a non-existent queue
:   Fixed a bug where referencing a non-existent `LocalQueue via kueue.x-k8s.io/queue-name` could cause a running pod to be terminated and permanently stuck with unremovable scheduling gates.

    ([OCPBUGS-78789](https://redhat.atlassian.net/browse/OCPBUGS-78789))

## Release notes for Red Hat build of Kueue version 1.3 {#release-notes-1.3_release-notes}

Red Hat build of Kueue version 1.3 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.3 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.16.

### New features and enhancements {#release-notes-1.3-new-features_release-notes}

Leader Worker Set Operator
:   Red Hat build of Kueue version 1.3 provides for the integration of the Leader Worker Set Operator with Red Hat build of Kueue so you can leverage the Red Hat build of Kueue scheduling and resource management functionality when running LeaderWorkerSets. For more information, see [Integrating the Leader Worker Set Operator](/openshift-docs-markdown/ai_workloads/kueue/integrating-lws#integrating-lws).

JobSet Operator
:   Red Hat build of Kueue version 1.3 provides for the integration of the JobSet Operator so you can use the JobSet Operator to manage and run large-scale, coordinated workloads like high-performance computing (HPC) and AI training. For more information, see [Integrating the JobSet Operator](/openshift-docs-markdown/ai_workloads/kueue/integrating-jobset#integrating-jobset).

Upstream progression of the Red Hat build of Kueue API to `v1beta2`
:   Red Hat build of Kueue version 1.3 provides the `v1beta2` version of the Red Hat build of Kueue API. This update continues the evolution of the Red Hat build of Kueue APIs with the ultimate goal of graduating the API to `v1`.

    All new Kueue objects created after the upgrade will be stored using the `v1beta2` version. The earlier version of the API, `v1beta1` is deprecated. Objects can still be created using `v1beta1`, if necessary. In these cases, a deprecation message is shown.

    However, existing objects are only auto-converted to the new storage version by Kubernetes during a write request. This means that Red Hat build of Kueue API objects that rarely receive updates such as Topologies, ResourceFlavors, or long-running Workloads could remain in the older `v1beta1` format indefinitely.

### Fixed issues {#release-notes-1.3-fixed-issues_release-notes}

Reconcile jobs only in opt-in namespaces
:   OpenShift Container Platform allowed reconciliation of `Job` resources that have the `kueue.x-k8s.io/queue-name` label, even if these resources are in namespaces that are not configured to opt in to being managed by OpenShift Container Platform. With this release, there is ongoing upstream work that updates this behavior so that Jobs with queue-name labels are also ignored unless their namespace matches the `managedJobsNamespaceSelector`. This change makes Red Hat build of Kueue behavior consistent across all integrations.

    ([OCPBUGS-58205](https://issues.redhat.com/browse/OCPBUGS-58205))

`Kueue` CR description reads as "Not available" in the OpenShift Container Platform web console
:   After installing Red Hat build of Kueue, in the **Operator details** view, the description for the `Kueue` CR read as "Not available". This issue did not affect or degrade the Red Hat build of Kueue Operator functionality. With this release, the "Not available" message no longer displays.

    ([OCPBUGS-62185](https://issues.redhat.com/browse/OCPBUGS-62185))

LeaderWorkerSet and Jobset validation errors
:   Currently, the Leader Worker Set Operator and JobSet Operator are only validated after the Operand CR is updated and the full Kueue hierarchy (ResourceFlavor, ClusterQueue, and LocalQueue) is established. Any configuration errors appear only when applying a LeaderWorkerSet or JobSet template.

    ([OCPBUGS-74210](https://issues.redhat.com/browse/OCPBUGS-74210))

### Known issues {#release-notes-1.3-known-issues_release-notes}

LeaderWorkerSet pods update sequentially by default
:   If you have integrated Leader Worker Set Operator with your Red Hat build of Kueue installation and you are using the rollout strategy option for updating LeaderWorkerSet pods, be aware that the `MaxUnavailable` feature gate in OpenShift Container Platform is disabled by default.

    When any change is made to LeaderWorkerSet pods, a rolling update is triggered. This action gradually replaces the old pods of a deployment with new ones, keeping as many pods alive as possible to avoid downtime. If `MaxUnavailable` is disabled, which is the OpenShift Container Platform default setting, the pods are updated one at a time.

    If you want to run updates in parallel instead of running them sequentially, `MaxUnavailable` feature gate must be enabled. For more information, see [Enabling feature sets at installation](/openshift-docs-markdown/nodes/clusters/nodes-cluster-enabling-features#nodes-cluster-enabling-features-install_nodes-cluster-enabling) and [Rollout Strategy](https://lws.sigs.k8s.io/docs/concepts/rollout-strategy/).

## Release notes for Red Hat build of Kueue version 1.2 {#release-notes-1.2_release-notes}

Red Hat build of Kueue version 1.2 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.2 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.14.

### New features and enhancements {#release-notes-1.2-new-features_release-notes}

Monitoring of pending workloads
:   Red Hat build of Kueue version 1.2 provides the `VisibilityOnDemand` feature to monitor the pipeline of pending jobs in the cluster queue and the local queue, and help users to estimate when their jobs will start. For more information, see [Monitoring pending workloads](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue#monitoring-pending-workloads-install-kueue).

### Fixed issues {#release-notes-1.2-fixed-issues_release-notes}

Custom resources are not deleted properly when you uninstall Red Hat build of Kueue
:   After you uninstall the Red Hat Build of Kueue Operator using the **Delete all operand instances for this operator** option in the OpenShift Container Platform web console, Red Hat build of Kueue custom resources were attempted to be deleted. With this release, they are not considered for deletion.

    ([OCPBUGS-62254](https://issues.redhat.com/browse/OCPBUGS-62254))

Documentation error in previous versions of Red Hat build of Kueue
:   In [Creating a Kueue custom resource](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue#create-kueue-cr_install-kueue), the optional workload types `Pod`, `Deployment`, `StatefulSet` were omitted. They are now included.

    ([OCPBUGS-62877](https://issues.redhat.com/browse/OCPBUGS-62877))

Red Hat build of Kueue metrics were not being exposed to Prometheus from version 1.1
:   Prometheus was not scraping metrics from the Operator’s controller, even though the ServiceMonitor and RBAC resources were successfully created as part of the Operator installation. As a result, none of the Kueue metrics were available in the cluster monitoring stack.

    The metrics service added during the installation was configured with an incorrect port reference, causing Prometheus to fail in scraping metrics from the Kueue endpoint. The port name has been updated with the correct port name.

    ([OCPBUGS-63441](https://issues.redhat.com/browse/OCPBUGS-63441))

### Known issues {#release-notes-1.2-known-issues_release-notes}

Reconcile jobs only in opt-in namespaces
:   OpenShift Container Platform allows reconciliation of `Job` resources that have the `kueue.x-k8s.io/queue-name` label, even if these resources are in namespaces which are not configured to opt in to being managed by OpenShift Container Platform. This is inconsistent with the behavior for other core integrations like pods, deployments, and stateful sets, which are only reconciled if they are in namespaces which have been configured to opt in to being managed by OpenShift Container Platform by adding the `kueue.openshift.io/managed=true`.

    ([OCPBUGS-58205](https://issues.redhat.com/browse/OCPBUGS-58205))

`Kueue` CR description reads as "Not available" in the OpenShift Container Platform web console
:   After installing Red Hat build of Kueue, in the **Operator details** view, the description for the `Kueue` CR reads as "Not available". This issue does not affect or degrade the Red Hat build of Kueue Operator functionality.

    ([OCPBUGS-62185](https://issues.redhat.com/browse/OCPBUGS-62185))

## Release notes for Red Hat build of Kueue version 1.1 {#kueue-release-notes-1.1_release-notes}

Red Hat build of Kueue version 1.1 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and later. Red Hat build of Kueue version 1.1 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.12.

> [!IMPORTANT]
> If you have a previously installed version of Red Hat build of Kueue on your cluster, you must uninstall the Operator and manually install version 1.1. For information see [Upgrading Red Hat build of Kueue](/openshift-docs-markdown/ai_workloads/kueue/install-kueue#upgrading-kueue_install-kueue).

### New features and enhancements {#release-notes-1.1-new-features_release-notes}

Configure a default local queue
:   A default local queue serves as the local queue for newly created jobs that do not have the `kueue.x-k8s.io/queue-name` label. After you create a default local queue, any new jobs created in the namespace without a `kueue.x-k8s.io/queue-name` label automatically update to have the `kueue.x-k8s.io/queue-name: default` label.

    ([RFE-7615](https://issues.redhat.com/browse/RFE-7615))

Multi-architecture and Hosted control planes support
:   With this release, Red Hat build of Kueue is supported on multiple different architectures, including ARM64, 64-bit x86, ppc64le (IBM Power(R)), and s390x (IBM Z(R)), as well as on Hosted control planes for OpenShift Container Platform.

    ([OCPSTRAT-2103](https://issues.redhat.com/browse/OCPSTRAT-2103))

    ([OCPSTRAT-2106](https://issues.redhat.com/browse/OCPSTRAT-2106))

### Fixed issues {#release-notes-1.1-fixed-issues_release-notes}

You can create a `Kueue` custom resource by using the OpenShift Container Platform web console
:   Before this update, if you tried to use the OpenShift Container Platform web console to create a `Kueue` custom resource (CR) by using the form view, the web console showed an error and the resource could not be created. With this release, the default namespace was removed from the `Kueue` CR template. As a result, you can use the OpenShift Container Platform web console to create a `Kueue` CR by using the form view.

    ([OCPBUGS-58118](https://issues.redhat.com/browse/OCPBUGS-58118))

### Known issues {#release-notes-1.1-known-issues_release-notes}

`Kueue` CR description reads as "Not available" in the OpenShift Container Platform web console
:   After you install Red Hat build of Kueue, in the **Operator details** view, the description for the `Kueue` CR reads as "Not available". This issue does not affect or degrade the Red Hat build of Kueue Operator functionality.

    ([OCPBUGS-62185](https://issues.redhat.com/browse/OCPBUGS-62185))

Custom resources are not deleted properly when you uninstall Red Hat build of Kueue
:   After you uninstall the Red Hat Build of Kueue Operator using the **Delete all operand instances for this operator** option in the OpenShift Container Platform web console, some Red Hat build of Kueue custom resources are not fully deleted. These resources can be viewed in the **Installed Operators** view with the status **Resource is being deleted**. As a workaround, you can manually delete the resource finalizers to remove them fully.

    ([OCPBUGS-62254](https://issues.redhat.com/browse/OCPBUGS-62254))

## Release notes for Red Hat build of Kueue version 1.0.1 {#release-notes-1.0.1_release-notes}

Red Hat build of Kueue version 1.0.1 is a patch release that is supported on OpenShift Container Platform versions 4.18 and 4.19 on the 64-bit x86 architecture.

Red Hat build of Kueue version 1.0.1 uses Kueue version 0.11.

### Bug fixes in Red Hat build of Kueue version 1.0.1 {#release-notes-1.0.1-bug-fixes_release-notes}

- Previously, leader election for Red Hat build of Kueue was not configured to tolerate disruption, which resulted in frequent crashing. With this release, the leader election values for Red Hat build of Kueue have been updated to match the durations recommended for OpenShift Container Platform. ([OCPBUGS-58496](https://issues.redhat.com/browse/OCPBUGS-58496))
- Previously, the `ReadyReplicas` count was not set in the reconciler, which meant that the Red Hat build of Kueue Operator status would report that there were no replicas ready. With this release, the `ReadyReplicas` count is based on the number of ready replicas for the deployment, which ensures that the Operator shows as ready in the OpenShift Container Platform console when the `kueue-controller-manager` pods are ready. ([OCPBUGS-59261](https://issues.redhat.com/browse/OCPBUGS-59261))
- Previously, when the `Kueue` custom resource (CR) was deleted from the `openshift-kueue-operator` namespace, the `kueue-manager-config` config map was not deleted automatically and could remain in the namespace. With this release, the `kueue-manager-config` config map, `kueue-webhook-server-cert` secret, and `metrics-server-cert` secret are deleted automatically when the `Kueue` CR is deleted. ([OCPBUGS-57960](https://issues.redhat.com/browse/OCPBUGS-57960))

## Release notes for Red Hat build of Kueue version 1.0 {#release-notes-1.0_release-notes}

Red Hat build of Kueue version 1.0 is a generally available release that is supported on OpenShift Container Platform versions 4.18 and 4.19 on the 64-bit x86 architecture. Red Hat build of Kueue version 1.0 uses [Kueue](https://kueue.sigs.k8s.io/docs/overview/) version 0.11.

### New features and enhancements {#release-notes-1.0-new-features_release-notes}

Role-based access control (RBAC)
:   Role-based access control (RBAC) enables you to control which types of users can create which types of Red Hat build of Kueue resources.

Configure resource quotas
:   Configuring resource quotas by creating cluster queues, resource flavors, and local queues enables you to control the amount of resources used by user-submitted jobs and workloads.

Control job and workload management
:   Labeling namespaces and configuring label policies enable you to control which jobs and workloads are managed by Red Hat build of Kueue.

Share borrowable resources between queues
:   Configuring cohorts, fair sharing, and gang scheduling settings enable you to share unused, borrowable resources between queues.

### Known issues {#release-notes-1.0-known-issues_release-notes}

Jobs in all namespaces are reconciled if they have the `kueue.x-k8s.io/queue-name` label
:   Red Hat build of Kueue uses the `managedJobsNamespaceSelector` configuration field, so that administrators can configure which namespaces opt in to be managed by Red Hat build of Kueue. Because namespaces must be manually configured to opt in to being managed by Red Hat build of Kueue, resources in system or third-party namespaces are not impacted or managed by Red Hat build of Kueue.

    The behavior in Red Hat build of Kueue 1.0 allows reconciliation of `Job` resources that have the `kueue.x-k8s.io/queue-name` label, even if these resources are in namespaces that are not configured to opt in to being managed by Red Hat build of Kueue. This is inconsistent with the behavior for other core integrations like pods, deployments, and stateful sets, which are only reconciled if they are in namespaces that have been configured to opt in to being managed by Red Hat build of Kueue.

    ([OCPBUGS-58205](https://issues.redhat.com/browse/OCPBUGS-58205))

You cannot create a `Kueue` custom resource by using the OpenShift Container Platform web console
:   If you try to use the OpenShift Container Platform web console to create a `Kueue` custom resource (CR) by using the form view, the web console shows an error and the resource cannot be created. As a workaround, use the YAML view to create a `Kueue` CR instead.

    ([OCPBUGS-58118](https://issues.redhat.com/browse/OCPBUGS-58118))

**Additional resources**
{._additional-resources}

- [Kueue (upstream documentation)](https://kueue.sigs.k8s.io/docs/overview/)