A workload is an application that will run to completion. It can be composed by one or multiple Pods that, loosely or tightly coupled, that, as a whole, complete a task. A workload is the unit of admission in Kueue.
The prototypical workload can be represented with a
For this reason, we sometimes use the word job to refer to any workload, and
Job when we refer specifically to the Kubernetes API.
However, Kueue does not directly manipulate Job objects. Instead, Kueue manages Workload objects that represent the resource requirements of an arbitrary workload. Kueue automatically creates a Workload for each Job object and syncs the decisions and statuses.
The manifest for a Workload looks like the following:
apiVersion: kueue.x-k8s.io/v1beta1 kind: Workload metadata: name: sample-job namespace: team-a spec: queueName: team-a-queue podSets: - count: 3 name: main template: spec: containers: - image: gcr.io/k8s-staging-perf-tests/sleep:latest imagePullPolicy: Always name: container resources: requests: cpu: "1" memory: 200Mi restartPolicy: Never
To indicate in which LocalQueue you want your Workload to be
enqueued, set the name of the LocalQueue in the
A Workload might be composed of multiple Pods with different pod specs.
Each item of the
.spec.podSets list represents a set of homogeneous Pods and has
the following fields:
specdescribes the pods using a
countis the number of pods that use the same
nameis a human-readable identifier for the pod set. You can use the role of the Pods in the Workload, like
Kueue uses the
podSets resources requests to calculate the quota used by a Workload and decide if and when to admit a Workload.
Kueue calculates the total resources usage for a Workload as the sum of the resource requests for each
podSet. The resource usage of a
podSet is equal to the resource requests of the pod spec multiplied by the
Requests values adjustment
Depending on the cluster setup, Kueue will adjust the resource usage of a Workload based on:
- The cluster defines default values in Limit Ranges, the default values will be used if not provided in the
- The created pods are subject of a Runtime Class Overhead.
- The spec defines only resource limits, case in which the limit values will be treated as requests.
Requests values validation
In cases when the cluster defines Limit Ranges, the values resulting from the adjustment above will be validated against the ranges.
Kueue will mark the workload as
Inadmissible if the range validation fails.
Reserved resource names
In addition to the usual resource naming restrictions, you cannot use the
pods resource name in a Pod spec, as it is reserved for internal Kueue use. You can use the
pods resource name in a ClusterQueue to set quotas on the maximum number of pods.
Workloads have a priority that influences the order in which they are admitted by a ClusterQueue. There are two ways to set the Workload priority:
Pod Priority: You can see the priority of the Workload in the field
.spec.priority. For a
batch/v1.Job, Kueue sets the priority of the Workload based on the pod priority of the Job’s pod template.
WorkloadPriority: Sometimes developers would like to control workload’s priority without affecting pod’s priority. By using
WorkloadPriority, you can independently manage the priority of workloads for queuing and preemption, separate from pod’s priority.
As described previously, Kueue has built-in support for workloads created with the Job API. But any custom workload API can integrate with Kueue by creating a corresponding Workload object for it.
It’s a mechanism allowing a currently Admitted workload to release a part of it’s Quota Reservation that is no longer needed.
Job integrations communicate this information by setting the
reclaimablePods status field, enumerating the number of pods per podset for which the Quota Reservation is no longer needed.
status: reclaimablePods: - name: podset1 count: 2 - name: podset2 count: 2
count can only increase while the workload holds a Quota Reservation.