Kueue API

value represents the integer value of this workloadPriorityClass. This is the actual priority that workloads receive when jobs have the name of this class in their workloadPriorityClass label. Changing the value of workloadPriorityClass doesn't affect the priority of workloads that were already created.

description is an arbitrary string that usually provides guidelines on when this workloadPriorityClass should be used.

clusterQueue is the name of the ClusterQueue that admitted this workload.

PodSetAssignments hold the admission results for each of the .spec.podSets entries.

ApiGroup is the group for the resource being referenced.

Kind is the type of the resource being referenced.

Name is the name of the resource being referenced.

controllerName identifies the controller that processes the AdmissionCheck, not necessarily a Kubernetes Pod or Deployment name. Cannot be empty.

RetryDelayMinutes specifies how long to keep the workload suspended after a failed check (after it transitioned to False). When the delay period has passed, the check state goes to "Unknown". The default is 15 min. Deprecated: retryDelayMinutes has already been deprecated since v0.8 and will be removed in v1beta2.

Parameters identifies a configuration with additional parameters for the check.

name identifies the admission check.

state of the admissionCheck, one of Pending, Ready, Retry, Rejected

lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.

message is a human readable message indicating details about the transition. This may be an empty string.

conditions hold the latest available observations of the AdmissionCheck current state.

name is an AdmissionCheck's name.

onFlavors is a list of ResourceFlavors' names that this AdmissionCheck should run for. If empty, the AdmissionCheck will run for all workloads submitted to the ClusterQueue.

admissionChecks is a list of strategies for AdmissionChecks

policy determines the policy for preemption to reclaim quota within cohort while borrowing. Possible values are:

• Never (default): do not allow for preemption, in other ClusterQueues within the cohort, for a borrowing workload.
• LowerPriority: allow preemption, in other ClusterQueues within the cohort, for a borrowing workload, but only if the preempted workloads are of lower priority.

maxPriorityThreshold allows to restrict the set of workloads which might be preempted by a borrowing workload, to only workloads with priority less than or equal to the specified threshold priority. When the threshold is not specified, then any workload satisfying the policy can be preempted by the borrowing workload.

Name indicates the name of the pending workload.

Namespace indicates the name of the pending workload.

Head contains the list of top pending workloads.

LastChangeTime indicates the time of the last change of the structure.

reclaimWithinCohort determines whether a pending Workload can preempt Workloads from other ClusterQueues in the cohort that are using more than their nominal quota. The possible values are:

• Never (default): do not preempt Workloads in the cohort.
• LowerPriority: Classic Preemption if the pending Workload fits within the nominal quota of its ClusterQueue, only preempt Workloads in the cohort that have lower priority than the pending Workload. Fair Sharing only preempt Workloads in the cohort that have lower priority than the pending Workload and that satisfy the fair sharing preemptionStategies.
• Any: Classic Preemption if the pending Workload fits within the nominal quota of its ClusterQueue, preempt any Workload in the cohort, irrespective of priority. Fair Sharing preempt Workloads in the cohort that satisfy the fair sharing preemptionStrategies.

borrowWithinCohort provides configuration to allow preemption within cohort while borrowing.

withinClusterQueue determines whether a pending Workload that doesn't fit within the nominal quota for its ClusterQueue, can preempt active Workloads in the ClusterQueue. The possible values are:

• Never (default): do not preempt Workloads in the ClusterQueue.
• LowerPriority: only preempt Workloads in the ClusterQueue that have lower priority than the pending Workload.
• LowerOrNewerEqualPriority: only preempt Workloads in the ClusterQueue that either have a lower priority than the pending workload or equal priority and are newer than the pending workload.

resourceGroups describes groups of resources. Each resource group defines the list of resources and a list of flavors that provide quotas for these resources. Each resource and each flavor can only form part of one resource group. resourceGroups can be up to 16.

cohort that this ClusterQueue belongs to. CQs that belong to the same cohort can borrow unused resources from each other.

A CQ can be a member of a single borrowing cohort. A workload submitted to a queue referencing this CQ can borrow quota from any CQ in the cohort. Only quota for the [resource, flavor] pairs listed in the CQ can be borrowed. If empty, this ClusterQueue cannot borrow from any other ClusterQueue and vice versa.

A cohort is a name that links CQs together, but it doesn't reference any object.

Validation of a cohort name is equivalent to that of object names: subdomain in DNS (RFC 1123).

QueueingStrategy indicates the queueing strategy of the workloads across the queues in this ClusterQueue. Current Supported Strategies:

• StrictFIFO: workloads are ordered strictly by creation time. Older workloads that can't be admitted will block admitting newer workloads even if they fit available quota.
• BestEffortFIFO: workloads are ordered by creation time, however older workloads that can't be admitted will not block admitting newer workloads that fit existing quota.

namespaceSelector defines which namespaces are allowed to submit workloads to this clusterQueue. Beyond this basic support for policy, a policy agent like Gatekeeper should be used to enforce more advanced policies. Defaults to null which is a nothing selector (no namespaces eligible). If set to an empty selector {}, then all namespaces are eligible.

flavorFungibility defines whether a workload should try the next flavor before borrowing or preempting in the flavor being evaluated.

preemption describes policies to preempt Workloads from this ClusterQueue or the ClusterQueue's cohort.

Preemption can happen in two scenarios:

• When a Workload fits within the nominal quota of the ClusterQueue, but the quota is currently borrowed by other ClusterQueues in the cohort. Preempting Workloads in other ClusterQueues allows this ClusterQueue to reclaim its nominal quota.
• When a Workload doesn't fit within the nominal quota of the ClusterQueue and there are admitted Workloads in the ClusterQueue with lower priority.

The preemption algorithm tries to find a minimal set of Workloads to preempt to accomomdate the pending Workload, preempting Workloads with lower priority first.

admissionChecks lists the AdmissionChecks required by this ClusterQueue. Cannot be used along with AdmissionCheckStrategy.

admissionCheckStrategy defines a list of strategies to determine which ResourceFlavors require AdmissionChecks. This property cannot be used in conjunction with the 'admissionChecks' property.

stopPolicy - if set to a value different from None, the ClusterQueue is considered Inactive, no new reservation being made.

Depending on its value, its associated workloads will:

• None - Workloads are admitted
• HoldAndDrain - Admitted workloads are evicted and Reserving workloads will cancel the reservation.
• Hold - Admitted workloads will run to completion and Reserving workloads will cancel the reservation.

fairSharing defines the properties of the ClusterQueue when participating in fair sharing. The values are only relevant if fair sharing is enabled in the Kueue configuration.

flavorsReservation are the reserved quotas, by flavor, currently in use by the workloads assigned to this ClusterQueue.

flavorsUsage are the used quotas, by flavor, currently in use by the workloads admitted in this ClusterQueue.

pendingWorkloads is the number of workloads currently waiting to be admitted to this clusterQueue.

reservingWorkloads is the number of workloads currently reserving quota in this clusterQueue.

admittedWorkloads is the number of workloads currently admitted to this clusterQueue and haven't finished yet.

conditions hold the latest available observations of the ClusterQueue current state.

PendingWorkloadsStatus contains the information exposed about the current status of the pending workloads in the cluster queue. Deprecated: This field will be removed on v1beta2, use VisibilityOnDemand (https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand/) instead.

FairSharing contains the information about the current status of fair sharing.

weight gives a comparative advantage to this ClusterQueue when competing for unused resources in the cohort against other ClusterQueues. The share of a ClusterQueue is based on the dominant resource usage above nominal quotas for each resource, divided by the weight. Admission prioritizes scheduling workloads from ClusterQueues with the lowest share and preempting workloads from the ClusterQueues with the highest share. A zero weight implies infinite share value, meaning that this ClusterQueue will always be at disadvantage against other ClusterQueues.

WeightedShare represent the maximum of the ratios of usage above nominal quota to the lendable resources in the cohort, among all the resources provided by the ClusterQueue, and divided by the weight. If zero, it means that the usage of the ClusterQueue is below the nominal quota. If the ClusterQueue has a weight of zero, this will return 9223372036854775807, the maximum possible share value.

whenCanBorrow determines whether a workload should try the next flavor before borrowing in current flavor. The possible values are:

• Borrow (default): allocate in current flavor if borrowing is possible.
• TryNextFlavor: try next flavor even if the current flavor has enough resources to borrow.

whenCanPreempt determines whether a workload should try the next flavor before borrowing in current flavor. The possible values are:

• Preempt: allocate in current flavor if it's possible to preempt some workloads.
• TryNextFlavor (default): try next flavor even if there are enough candidates for preemption in the current flavor.

name of this flavor. The name should match the .metadata.name of a ResourceFlavor. If a matching ResourceFlavor does not exist, the ClusterQueue will have an Active condition set to False.

resources is the list of quotas for this flavor per resource. There could be up to 16 resources.

name of the flavor.

resources lists the quota usage for the resources in this flavor.

Location of the KubeConfig.

If LocationType is Secret then Location is the name of the secret inside the namespace in which the kueue controller manager is running. The config should be stored in the "kubeconfig" key.

Type of the KubeConfig location.

name of the flavor.

resources used in the flavor.

nodeLabels are labels that associate the ResourceFlavor with Nodes that have the same labels.

nodeTaints are taints that the nodes associated with this ResourceFlavor have.

name of the flavor.

resources lists the quota usage for the resources in this flavor.

name of the resource.

total is the total quantity of used quota.

clusterQueue is a reference to a clusterQueue that backs this localQueue.

stopPolicy - if set to a value different from None, the LocalQueue is considered Inactive, no new reservation being made.

Depending on its value, its associated workloads will:

• None - Workloads are admitted
• HoldAndDrain - Admitted workloads are evicted and Reserving workloads will cancel the reservation.
• Hold - Admitted workloads will run to completion and Reserving workloads will cancel the reservation.

PendingWorkloads is the number of Workloads in the LocalQueue not yet admitted to a ClusterQueue

reservingWorkloads is the number of workloads in this LocalQueue reserving quota in a ClusterQueue and that haven't finished yet.

admittedWorkloads is the number of workloads in this LocalQueue admitted to a ClusterQueue and that haven't finished yet.

Conditions hold the latest available observations of the LocalQueue current state.

flavorsReservation are the reserved quotas, by flavor currently in use by the workloads assigned to this LocalQueue.

flavorsUsage are the used quotas, by flavor currently in use by the workloads assigned to this LocalQueue.

flavors lists all currently available ResourceFlavors in specified ClusterQueue.

Information how to connect to the cluster.

List of MultiKueueClusters names where the workloads from the ClusterQueue should be distributed.

name is the PodSet name.

template is the Pod template.

The only allowed fields in template.metadata are labels and annotations.

If requests are omitted for a container or initContainer, they default to the limits if they are explicitly specified for the container or initContainer.

During admission, the rules in nodeSelector and nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution that match the keys in the nodeLabels from the ResourceFlavors considered for this Workload are used to filter the ResourceFlavors that can be assigned to this podSet.

count is the number of pods for the spec.

minCount is the minimum number of pods for the spec acceptable if the workload supports partial admission.

If not provided, partial admission for the current PodSet is not enabled.

Only one podSet within the workload can use this.

This is an alpha field and requires enabling PartialAdmission feature gate.

topologyRequest defines the topology request for the PodSet.

Name is the name of the podSet. It should match one of the names in .spec.podSets.

Flavors are the flavors assigned to the workload for each resource.

resourceUsage keeps track of the total resources all the pods in the podset need to run.

Beside what is provided in podSet's specs, this calculation takes into account the LimitRange defaults and RuntimeClass overheads at the moment of admission. This field will not change in case of quota reclaim.

count is the number of pods taken into account at admission time. This field will not change in case of quota reclaim. Value could be missing for Workloads created before this field was added, in that case spec.podSets[*].count value will be used.

topologyAssignment indicates the topology assignment divided into topology domains corresponding to the lowest level of the topology. The assignment specifies the number of Pods to be scheduled per topology domain and specifies the node selectors for each topology domain, in the following way: the node selector keys are specified by the levels field (same for all domains), and the corresponding node selector value is specified by the domains.values subfield. If the TopologySpec.Levels field contains "kubernetes.io/hostname" label, topologyAssignment will contain data only for this label, and omit higher levels in the topology

Example:

topologyAssignment: levels:

• cloud.provider.com/topology-block
• cloud.provider.com/topology-rack domains:
• values: [block-1, rack-1] count: 4
• values: [block-1, rack-2] count: 2

Here:

• 4 Pods are to be scheduled on nodes matching the node selector: cloud.provider.com/topology-block: block-1 cloud.provider.com/topology-rack: rack-1
• 2 Pods are to be scheduled on nodes matching the node selector: cloud.provider.com/topology-block: block-1 cloud.provider.com/topology-rack: rack-2

Example: Below there is an equivalent of the above example assuming, Topology object defines kubernetes.io/hostname as the lowest level in topology. Hence we omit higher level of topologies, since the hostname label is sufficient to explicitly identify a proper node.

topologyAssignment: levels:

• kubernetes.io/hostname domains:
• values: [hostname-1] count: 4
• values: [hostname-2] count: 2

name is the name of the podSet. It should match one of the names in .spec.podSets.

resources is the total resources all the pods in the podset need to run.

Beside what is provided in podSet's specs, this value also takes into account the LimitRange defaults and RuntimeClass overheads at the moment of consideration and the application of resource.excludeResourcePrefixes and resource.transformations.

required indicates the topology level required by the PodSet, as indicated by the kueue.x-k8s.io/podset-required-topology PodSet annotation.

preferred indicates the topology level preferred by the PodSet, as indicated by the kueue.x-k8s.io/podset-preferred-topology PodSet annotation.

PodIndexLabel indicates the name of the label indexing the pods. For example, in the context of

• kubernetes job this is: kubernetes.io/job-completion-index
• JobSet: kubernetes.io/job-completion-index (inherited from Job)
• Kubeflow: training.kubeflow.org/replica-index

SubGroupIndexLabel indicates the name of the label indexing the instances of replicated Jobs (groups) within a PodSet. For example, in the context of JobSet this is jobset.sigs.k8s.io/job-index.

SubGroupIndexLabel indicates the count of replicated Jobs (groups) within a PodSet. For example, in the context of JobSet this value is read from jobset.sigs.k8s.io/replicatedjob-replicas.

Name of the PodSet to modify. Should match to one of the Workload's PodSets.

ProvisioningClassName describes the different modes of provisioning the resources. Check autoscaling.x-k8s.io ProvisioningRequestSpec.ProvisioningClassName for details.

Parameters contains all other parameters classes may require.

managedResources contains the list of resources managed by the autoscaling.

If empty, all resources are considered managed.

If not empty, the ProvisioningRequest will contain only the podsets that are requesting at least one of them.

If none of the workloads podsets is requesting at least a managed resource, the workload is considered ready.

retryStrategy defines strategy for retrying ProvisioningRequest. If null, then the default configuration is applied with the following parameter values: backoffLimitCount: 3 backoffBaseSeconds: 60 - 1 min backoffMaxSeconds: 1800 - 30 mins

To switch off retry mechanism set retryStrategy.backoffLimitCount to 0.

BackoffLimitCount defines the maximum number of re-queuing retries. Once the number is reached, the workload is deactivated (.spec.activate=false).

Every backoff duration is about "b*2^(n-1)+Rand" where:

• "b" represents the base set by "BackoffBaseSeconds" parameter,
• "n" represents the "workloadStatus.requeueState.count",
• "Rand" represents the random jitter. During this time, the workload is taken as an inadmissible and other workloads will have a chance to be admitted. By default, the consecutive requeue delays are around: (60s, 120s, 240s, ...).

Defaults to 3.

BackoffBaseSeconds defines the base for the exponential backoff for re-queuing an evicted workload.

Defaults to 60.

BackoffMaxSeconds defines the maximum backoff time to re-queue an evicted workload.

Defaults to 1800.

name is the PodSet name.

count is the number of pods for which the requested resources are no longer needed.

count records the number of times a workload has been re-queued When a deactivated (.spec.activate=false) workload is reactivated (.spec.activate=true), this count would be reset to null.

requeueAt records the time when a workload will be re-queued. When a deactivated (.spec.activate=false) workload is reactivated (.spec.activate=true), this time would be reset to null.

nodeLabels are labels that associate the ResourceFlavor with Nodes that have the same labels. When a Workload is admitted, its podsets can only get assigned ResourceFlavors whose nodeLabels match the nodeSelector and nodeAffinity fields. Once a ResourceFlavor is assigned to a podSet, the ResourceFlavor's nodeLabels should be injected into the pods of the Workload by the controller that integrates with the Workload object.

nodeLabels can be up to 8 elements.

nodeTaints are taints that the nodes associated with this ResourceFlavor have. Workloads' podsets must have tolerations for these nodeTaints in order to get assigned this ResourceFlavor during admission.

An example of a nodeTaint is cloud.provider.com/preemptible="true":NoSchedule

nodeTaints can be up to 8 elements.

tolerations are extra tolerations that will be added to the pods admitted in the quota associated with this resource flavor.

An example of a toleration is cloud.provider.com/preemptible="true":NoSchedule

tolerations can be up to 8 elements.

topologyName indicates topology for the TAS ResourceFlavor. When specified, it enables scraping of the topology information from the nodes matching to the Resource Flavor node labels.

coveredResources is the list of resources covered by the flavors in this group. Examples: cpu, memory, vendor.com/gpu. The list cannot be empty and it can contain up to 16 resources.

flavors is the list of flavors that provide the resources of this group. Typically, different flavors represent different hardware models (e.g., gpu models, cpu architectures) or pricing models (on-demand vs spot cpus). Each flavor MUST list all the resources listed for this group in the same order as the .resources field. The list cannot be empty and it can contain up to 16 flavors.

name of this resource.

nominalQuota is the quantity of this resource that is available for Workloads admitted by this ClusterQueue at a point in time. The nominalQuota must be non-negative. nominalQuota should represent the resources in the cluster available for running jobs (after discounting resources consumed by system components and pods not managed by kueue). In an autoscaled cluster, nominalQuota should account for resources that can be provided by a component such as Kubernetes cluster-autoscaler.

If the ClusterQueue belongs to a cohort, the sum of the quotas for each (flavor, resource) combination defines the maximum quantity that can be allocated by a ClusterQueue in the cohort.

borrowingLimit is the maximum amount of quota for the [flavor, resource] combination that this ClusterQueue is allowed to borrow from the unused quota of other ClusterQueues in the same cohort. In total, at a given time, Workloads in a ClusterQueue can consume a quantity of quota equal to nominalQuota+borrowingLimit, assuming the other ClusterQueues in the cohort have enough unused quota. If null, it means that there is no borrowing limit. If not null, it must be non-negative. borrowingLimit must be null if spec.cohort is empty.

lendingLimit is the maximum amount of unused quota for the [flavor, resource] combination that this ClusterQueue can lend to other ClusterQueues in the same cohort. In total, at a given time, ClusterQueue reserves for its exclusive use a quantity of quota equals to nominalQuota - lendingLimit. If null, it means that there is no lending limit, meaning that all the nominalQuota can be borrowed by other clusterQueues in the cohort. If not null, it must be non-negative. lendingLimit must be null if spec.cohort is empty. This field is in beta stage and is enabled by default.

name of the resource

total is the total quantity of used quota, including the amount borrowed from the cohort.

Borrowed is quantity of quota that is borrowed from the cohort. In other words, it's the used quota that is over the nominalQuota.

levels is an ordered list of keys denoting the levels of the assigned topology (i.e. node label keys), from the highest to the lowest level of the topology.

domains is a list of topology assignments split by topology domains at the lowest level of the topology.

values is an ordered list of node selector values describing a topology domain. The values correspond to the consecutive topology levels, from the highest to the lowest.

count indicates the number of Pods to be scheduled in the topology domain indicated by the values field.

podSets is a list of sets of homogeneous pods, each described by a Pod spec and a count. There must be at least one element and at most 8. podSets cannot be changed.

queueName is the name of the LocalQueue the Workload is associated with. queueName cannot be changed while .status.admission is not null.

If specified, indicates the workload's priority. "system-node-critical" and "system-cluster-critical" are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the workload priority will be default or zero if there is no default.

Priority determines the order of access to the resources managed by the ClusterQueue where the workload is queued. The priority value is populated from PriorityClassName. The higher the value, the higher the priority. If priorityClassName is specified, priority must not be null.

priorityClassSource determines whether the priorityClass field refers to a pod PriorityClass or kueue.x-k8s.io/workloadpriorityclass. Workload's PriorityClass can accept the name of a pod priorityClass or a workloadPriorityClass. When using pod PriorityClass, a priorityClassSource field has the scheduling.k8s.io/priorityclass value.

Active determines if a workload can be admitted into a queue. Changing active from true to false will evict any running workloads. Possible values are:

• false: indicates that a workload should never be admitted and evicts running workloads
• true: indicates that a workload can be evaluated for admission into it's respective queue.

Defaults to true

maximumExecutionTimeSeconds if provided, determines the maximum time, in seconds, the workload can be admitted before it's automatically deactivated.

If unspecified, no execution time limit is enforced on the Workload.

admission holds the parameters of the admission of the workload by a ClusterQueue. admission can be set back to null, but its fields cannot be changed once set.

requeueState holds the re-queue state when a workload meets Eviction with PodsReadyTimeout reason.

conditions hold the latest available observations of the Workload current state.

The type of the condition could be:

• Admitted: the Workload was admitted through a ClusterQueue.
• Finished: the associated workload finished running (failed or succeeded).
• PodsReady: at least .spec.podSets[*].count Pods are ready or have succeeded.

reclaimablePods keeps track of the number pods within a podset for which the resource reservation is no longer needed.

admissionChecks list all the admission checks required by the workload and the current status

resourceRequests provides a detailed view of the resources that were requested by a non-admitted workload when it was considered for admission. If admission is non-null, resourceRequests will be empty because admission.resourceUsage contains the detailed information.

accumulatedPastExexcutionTimeSeconds holds the total time, in seconds, the workload spent in Admitted state, in the previous Admit - Evict cycles.