Resources in a cluster are typically not homogeneous. Resources could differ in:
- pricing and availability (for example, spot verus on-demand VMs)
- architecture (for example, x86 versus ARM CPUs)
- brands and models (for example, Radeon 7000 versus Nvidia A100 versus T4 GPUs)
A ResourceFlavor is an object that represents these resource variations and allows you to associate them with cluster nodes through labels, taints and tolerations.
Note: If the resources in your cluster are homogeneous, you can use an empty ResourceFlavor instead of adding labels to custom ResourceFlavors.
A sample ResourceFlavor looks like the following:
- effect: NoSchedule
- key: "spot-taint"
You can use the
.metadata.name field to reference a ResourceFlavor from a
ClusterQueue in the
Requires Kubernetes 1.23 or newer
To associate a ResourceFlavor with a subset of nodes of your cluster, you can
.spec.nodeLabels field with matching node labels that uniquely identify
the nodes. If you are using cluster autoscaler
(or equivalent controllers), make sure that the controller is configured to add those labels when
adding new nodes.
To guarantee that the Pods in the Workload run on the nodes associated to the flavor that Kueue selected, Kueue performs the following steps:
When admitting a Workload, Kueue evaluates the
.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecutionfields in the PodSpecs of your Workload against the ResourceFlavor labels.
ResourceFlavorsthat don’t match the node affinity of the Workload cannot be assigned to a Workload’s podSet.
Once the Workload is admitted:
Kueue adds the ResourceFlavor labels to the
.nodeSelectorof the underlying Workload Pod templates. This occurs if the Workload didn’t specify the
ResourceFlavorlabels already as part of its nodeSelector.
For example, for a batch/v1.Job, Kueue adds the labels to the
.spec.template.spec.nodeSelectorfield. This guarantees that the workload’s Pods can only be scheduled on the nodes targeted by the flavor that Kueue assigned to the Workload.
Kueue adds the tolerations to the underlying Workload Pod templates.
For example, for a batch/v1.Job, Kueue adds the tolerations to the
.spec.template.spec.tolerationsfield. This allows that the workloads Pods to be scheduled on nodes having specific taints.
To restrict the usage of a ResourceFlavor, you can configure the
These taints should typically match the taints of the Nodes associated with the ResourceFlavor.
Taints on the ResourceFlavor work similarly to Node taints. For Kueue to admit a Workload to use the ResourceFlavor, the PodSpecs in the Workload should have a toleration for it. As opposed to the behavior for ResourceFlavor labels, Kueue does not add tolerations for the flavor taints.
If your cluster has homogeneous resources, or if you don’t need to manage quotas for the different flavors of a resource separately, you can create a ResourceFlavor without any labels or taints. Such ResourceFlavor is called an empty ResourceFlavor and its definition looks like the following: