Run A JobSet
This document explains how you can use Kueue’s scheduling and resource management functionality when running JobSet Operator JobSet.
This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.
Before you begin
Check Administer cluster quotas for details on the initial Kueue setup.
See JobSet Installation for installation and configuration details of JobSet Operator.
Note
In order to use JobSet, prior to v0.8.1, you need to restart Kueue after the installation. You can do it by running:kubectl delete pods -l control-plane=controller-manager -n kueue-system.JobSet definition
When running JobSets on Kueue, take into consideration the following aspects:
a. Queue selection
The target local queue should be specified in the metadata.labels section of the JobSet configuration.
metadata:
labels:
kueue.x-k8s.io/queue-name: user-queue
b. Configure the resource needs
The resource needs of the workload can be configured in the spec.replicatedJobs. Should also be taken into account that number of replicas, parallelism and completions affect the resource calculations.
- replicas: 1
template:
spec:
completions: 2
parallelism: 2
template:
spec:
containers:
- resources:
requests:
cpu: 1
c. Jobs prioritisation
The first PriorityClassName of spec.replicatedJobs that is not empty will be used as the priority.
- template:
spec:
template:
spec:
priorityClassName: high-priority
d. Topology-Aware Scheduling
Note
The following examples require your cluster to be configured for Topology-Aware Scheduling. This includes having appropriate labels on your nodes, a configuredTopology object, and a ResourceFlavor referencing that topology. Please refer to the Topology-Aware Scheduling guide for full cluster setup instructions before running these examples.You can use Topology-Aware Scheduling (TAS) to schedule ReplicatedJobs within specific topology domains (like racks or zones) to minimize network latency. Kueue supports three TAS placement scenarios for JobSets using annotations on the Pod template:
1. Same-domain placement (Just PodSet TAS)
Force the entire ReplicatedJob to be scheduled in a single topology domain.
# sample-jobset-tas-required.yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: jobset-tas-required
labels:
kueue.x-k8s.io/queue-name: user-queue
spec:
replicatedJobs:
- name: workers
replicas: 1
template:
spec:
parallelism: 4
completions: 4
backoffLimit: 0
template:
metadata:
annotations:
kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-rack"
spec:
containers:
- name: sleep
image: busybox
resources:
requests:
cpu: 1
memory: "200Mi"
command:
- sleep
args:
- 100s
2. Sliced placement (Just PodSet-Slice TAS)
Split the ReplicatedJob into smaller slices, where each slice must fit in a single topology domain, but different slices can be in different domains.
# sample-jobset-tas-sliced.yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: jobset-tas-sliced
labels:
kueue.x-k8s.io/queue-name: user-queue
spec:
replicatedJobs:
- name: workers
replicas: 1
template:
spec:
parallelism: 8
completions: 8
backoffLimit: 0
template:
metadata:
annotations:
kueue.x-k8s.io/podset-slice-required-topology: "cloud.provider.com/topology-rack"
kueue.x-k8s.io/podset-slice-size: "4"
spec:
containers:
- name: sleep
image: busybox
resources:
requests:
cpu: 1
memory: "200Mi"
command:
- sleep
args:
- 100s
3. Combined constraint (Both PodSet and PodSet-Slice TAS)
Force the entire ReplicatedJob into a larger domain (e.g. block), but allow it to be sliced across smaller domains (e.g. rack) within that block. (Note: This is different from Kueue’s Multi-Layer Topology feature, which uses the podset-slice-required-topology-constraints annotation).
# sample-jobset-tas-combined.yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: jobset-tas-combined
labels:
kueue.x-k8s.io/queue-name: user-queue
spec:
replicatedJobs:
- name: workers
replicas: 1
template:
spec:
parallelism: 8
completions: 8
backoffLimit: 0
template:
metadata:
annotations:
kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-block"
kueue.x-k8s.io/podset-slice-required-topology: "cloud.provider.com/topology-rack"
kueue.x-k8s.io/podset-slice-size: "4"
spec:
containers:
- name: sleep
image: busybox
resources:
requests:
cpu: 1
memory: "200Mi"
command:
- sleep
args:
- 100s
Multikueue
Check the Multikueue for details on running Jobsets in MultiKueue environment.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.