Run Workloads on Kueue

Run A Kubernetes Job

Mon, 14 Feb 2022 00:00:00 +0000

This page shows you how to run a Job in a Kubernetes cluster with Kueue enabled.

The intended audience for this page are batch users.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
The cluster has quotas configured.

The following picture shows all the concepts you will interact with in this tutorial:

Run A CronJob

Tue, 12 Dec 2023 00:00:00 +0000

This page shows you how to run a CronJob in a Kubernetes cluster with Kueue enabled.

The intended audience for this page are batch users.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
The cluster has quotas configured.

0. Identify the queues available in your namespace

Run the following command to list the LocalQueues available in your namespace.

Using HAMi

Mon, 15 Dec 2025 00:00:00 +0000

This page demonstrates how to use Kueue for running workloads with HAMi (Heterogeneous AI Computing Virtualization Middleware) for vGPU resource management.

When working with vGPU resources through HAMi, Pods request per-vGPU resources (nvidia.com/gpumem, nvidia.com/gpucores) along with the number of vGPUs (nvidia.com/gpu). For quota management, you need to track the total resource consumption across all vGPU instances.

Below we demonstrate how to support this with the resource transformation for quota management in Kueue.

Run LeaderWorkerSet

Mon, 17 Feb 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running LeaderWorkerSet.

We demonstrate how to support scheduling LeaderWorkerSets where a group of Pods constitutes a unit of admission represented by a Workload. This allows to scale-up and down LeaderWorkerSets group by group.

This integration is based on the Plain Pod Group integration.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Run An AppWrapper

Wed, 08 Jan 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running AppWrappers.

AppWrappers provide a flexible and workload-agnostic mechanism for enabling Kueue to manage a group of Kubernetes resources as a single logical unit without requiring any Kueue-specific support by the controllers of those resources.

AppWrappers are designed to harden workloads by providing an additional level of automatic fault detection and recovery. The AppWrapper controller monitors the health of the workload and if corrective actions are not taken by the primary resource controllers within specified deadlines, the AppWrapper controller will orchestrate workload-level retries and resource deletion to ensure that either the workload returns to a healthy state or is cleanly removed from the cluster and its quota freed for use by other workloads.

Run A TrainJob

Tue, 05 Nov 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Kubeflow Trainer TrainJobs.

Kubeflow Trainer v2 introduces the TrainJob API that works seamlessly with Kueue for batch scheduling and resource management. TrainJobs can be configured to use either:

ClusterTrainingRuntime: Cluster-scoped training runtimes that can be used across all namespaces
TrainingRuntime: Namespace-scoped training runtimes that are only available within a specific namespace

Kueue manages TrainJobs by scheduling their underlying jobs according to available quota and priority.

Run Deployment

Thu, 25 Jul 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Deployments. Although Kueue does not yet support managing a Deployment as a single Workload, it’s still possible to leverage Kueue’s scheduling and resource management capabilities for the individual Pods of the Deployment.

We demonstrate how to support scheduling Deployments in Kueue based on the Plain Pod integration, where every Pod from a Deployment is represented as a single independent Plain Pod. This approach allows independent resource management for the Pods, and thus scale-out and scale-in of the Deployment.

Run StatefulSet

Thu, 25 Jul 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running StatefulSets.

We demonstrate how to support scheduling StatefulSets in Kueue based on the Plain Pod Group integration, where StatefulSet is represented as a single independent workload.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

The statefulset integration is enabled by default.

Run Plain Pods

Wed, 27 Sep 2023 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running plain Pods. Kueue supports management of both individual Pods, or Pod groups.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

The pod integration is enabled by default.

Run Workloads With DRA Devices

Sun, 22 Mar 2026 00:00:00 +0000

This page shows you how to run workloads that request hardware devices (such as GPUs) managed by Dynamic Resource Allocation (DRA) in a Kubernetes cluster with Kueue enabled. The examples use a batch Job, but the same approach works with any workload type that Kueue supports.

The intended audience for this page are batch users.

For conceptual details about how Kueue handles DRA resources, see Dynamic Resource Allocation concepts.

Before you begin

Make sure the following conditions are met:

Run Jobs Using Python

Wed, 05 Jul 2023 00:00:00 +0000

This guide is for batch users that have a basic understanding of interacting with Kubernetes from Python. For more information, see Kueue’s overview.

Before you begin

Check administer cluster quotas for details on the initial cluster setup. You’ll also need uv installed. Each example script includes inline script metadata (PEP 723) so uv run will automatically create an isolated environment and install the required dependencies.

Run A JobSet

Fri, 16 Jun 2023 00:00:00 +0000

This document explains how you can use Kueue’s scheduling and resource management functionality when running JobSet Operator JobSet.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Check Administer cluster quotas for details on the initial Kueue setup.

Run Workloads with Topology-Aware Scheduling

Sun, 17 May 2026 00:00:00 +0000

This page shows you how to run workloads that use Topology-Aware Scheduling (TAS) in a Kubernetes cluster with Kueue enabled. The examples use a batch Job, but the same annotations work with any workload type that Kueue supports.

This page is intended for batch users. If you are a batch administrator looking to configure cluster topology and queues, see Setup Topology-Aware Scheduling.

For conceptual details about how Kueue models cluster topology and how the TAS scheduling algorithm works, see Topology-Aware Scheduling concepts.

Run A RayService

Mon, 30 Jun 2025 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running RayService.

Kueue manages the RayService as a top-level job, similar to how Kueue manages the RayJob.

This guide is for serving users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Make sure you are using Kueue v0.17.0 or newer and KubeRay v1.3.0 or newer.

Run A RayCluster

Wed, 07 Aug 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running RayCluster.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Make sure you are using Kueue v0.6.0 version or newer and KubeRay v1.1.0 or newer.

Run A RayJob

Wed, 07 Aug 2024 00:00:00 +0000

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running KubeRay’s RayJob.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Make sure you are using Kueue v0.6.0 version or newer and KubeRay v1.1.0 or newer.