Topology Aware Scheduling

It is common that AI/ML workloads require a significant amount of pod-to-pod communication. Therefore the network bandwidth between the running Pods translates into the workload execution time, and the cost of running such workloads. The available bandwidth between the Pods depends on the placement of the Nodes, running the Pods, in the data center.

We observe that the data centers have a hierarchical structure of their organizational units, like racks and blocks, where there are multiple nodes within a rack, and there are multiple racks within a block. Pods running within the same organizational unit have better network bandwidth than Pods on different units. We say that nodes placed in different racks are more distant than nodes placed within the same rack. Similarly, nodes placed in different blocks are more distant than two nodes within the same block.

In this feature (called Topology Aware Scheduling, or TAS for short) we introduce a convention to represent the hierarchical node topology information, and a set of APIs for Kueue administrators and users to utilize the information to optimize the Pod placement.

Node topology information

We propose a lightweight model for representing the hierarchy of nodes within a data center by using node labels. In this model the node labels are set up by a cloud provider, or set up manually by administrators of on-premise clusters.

Additionally, we assume that every node used for TAS has a set of the labels which identifies uniquely its location in the tree structure. We do not assume global uniqueness of labels on each level, i.e. there could be two nodes with the same “rack” label, but in different “blocks”.

For example, this is a representation of the data center hierarchy;

Note that, there is a pair of nodes, node-1 and node-3, with the same value of the “cloud.provider.com/topology-rack” label, but in different blocks.

Capacity calculation

For each PodSet TAS determines the current free capacity per each topology domain (like a given rack) by:

• including Node allocatable capacity (based on the .status.allocatable field) of only ready (with Ready=True condition) and schedulable (with .spec.unschedulable=false) Nodes,
• subtracting the usage coming from all other admitted TAS workloads,
• subtracting the usage coming from all other non-TAS Pods (owned mainly by DaemonSets, but also including static Pods, Deployments, etc.).

Admin-facing APIs

As an admin, in order to enable the feature you need to:

1. create at least one instance of the Topology API
2. reference the Topology API from a dedicated ResourceFlavor by the .spec.topologyName field

Example

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: "default"
spec:
  levels:
  - nodeLabel: "cloud.provider.com/topology-block"
  - nodeLabel: "cloud.provider.com/topology-rack"
  - nodeLabel: "kubernetes.io/hostname"
---
kind: ResourceFlavor
apiVersion: kueue.x-k8s.io/v1beta2
metadata:
  name: "tas-flavor"
spec:
  nodeLabels:
    cloud.provider.com/node-group: "tas-group"
  topologyName: "default"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "tas-cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "tas-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 100
      - name: "memory"
        nominalQuota: 100Gi
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "tas-user-queue"
spec:
  clusterQueue: "tas-cluster-queue"

An example for managing GPUs:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: "default"
spec:
  levels:
  - nodeLabel: "cloud.provider.com/topology-block"
  - nodeLabel: "cloud.provider.com/topology-rack"
  - nodeLabel: "kubernetes.io/hostname" # Taints on nodes are only respected if this label is the lowest topology level.
---
kind: ResourceFlavor
apiVersion: kueue.x-k8s.io/v1beta2
metadata:
  name: "tas-flavor"
spec:
  nodeLabels:
    cloud.provider.com/node-group: "tas-group"
  topologyName: "default"
  tolerations:
  - key: "nvidia.com/gpu" # Most cloud providers auto inject the toleration to the pods based on the nodeSelector via a webhook. However, TAS isn't aware of the webhook, so you need to manually add the toleration here.
    operator: "Exists"
    effect: "NoSchedule"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "tas-cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: "tas-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 100
      - name: "memory"
        nominalQuota: 100Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 16
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "tas-user-queue"
spec:
  clusterQueue: "tas-cluster-queue"

User-facing APIs

Once TAS is configured and ready to be used, you can create Jobs with the following annotations set at the PodTemplate level:

• kueue.x-k8s.io/podset-preferred-topology - indicates that a PodSet requires Topology Aware Scheduling, but scheduling all pods within pods on nodes within the same topology domain is a preference rather than requirement. The levels are evaluated one-by-one going up from the level indicated by the annotation. If the PodSet cannot fit within a given topology domain then the next topology level up is considered. If the PodSet cannot fit at the highest topology level, then it gets admitted as distributed among multiple topology domains.
• kueue.x-k8s.io/podset-required-topology - indicates that a PodSet requires Topology Aware Scheduling, and requires scheduling all pods on nodes within the same topology domain corresponding to the topology level indicated by the annotation value (e.g. within a rack or within a block).
• kueue.x-k8s.io/podset-unconstrained-topology - indicates that a PodSet requires Topology Aware Scheduling, and requires scheduling all pods on any nodes without topology considerations. In other words, this considers if all pods could be accommodated within any nodes which helps to minimize fragmentation by filling the small gaps on nodes across the cluster.
• kueue.x-k8s.io/podset-group-name - indicates the name of the group of PodSets. PodSet Group is a unit of flavor assignment and topology domain fitting. This is useful when you want to ensure that multiple PodSets are scheduled in the same topology domain.
• kueue.x-k8s.io/podset-slice-required-topology-constraints - defines multi-layer slice topology constraints as a JSON-encoded array. Each entry specifies a topology level and slice size, ordered from the outermost (coarsest) to the innermost (finest) layer. At most 3 layers are supported. This annotation is mutually exclusive with kueue.x-k8s.io/podset-slice-required-topology and kueue.x-k8s.io/podset-slice-size. Requires the TASMultiLayerTopology feature gate.

Example

Here is an example Job a user might submit to use TAS. It assumes there exists a LocalQueue named tas-user-queue which refernces the ClusterQueue pointing to a TAS ResourceFlavor.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: tas-sample-preferred
  labels:
    kueue.x-k8s.io/queue-name: tas-user-queue
spec:
  parallelism: 40
  completions: 40
  completionMode: Indexed
  template:
    metadata:
      annotations:
        kueue.x-k8s.io/podset-preferred-topology: "cloud.provider.com/topology-block"
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
      restartPolicy: Never

ClusterAutoscaler support

TAS integrates with the Kubernetes ClusterAutoscaler through the Provisioning AdmissionCheck.

When a workload is assigned to the TAS ResourceFlavor with Provisioning AdmissionCheck, then its admission flow has the following stages:

1. Quota reservation: quota is reserved, and the Workload obtains the QuotaReserved condition. Preemptions are evaluated if configured.
2. Admission checks: Kueue waits for all AdmissionChecks, including the Provisioning one, to report Ready inside the Workload’s status.admissionChecks field.
3. Topology assignment: Kueue sets topology assignment, on the Workload object, calculated taking into account any newly provisioned nodes.

Check also PodSet updates in ProvisioningRequestConfig to see how you can configure Kueue if you want to restrict scheduling to the newly provisioned nodes (assuming the provisioning class supports it).

Hot swap support

When the lowest level of Topology is set to node, TAS finds a fixed assignment of pods to nodes and injects a NodeSelector to make sure the pods get scheduled on the selected nodes. But this means that in case of any node failures or deletions, which occur during the runtime of a workload, the workload cannot run on any other nodes. In order to avoid costly re-scheduling of the entire TAS workload we introduce the node hot swap feature.

With this feature, TAS tries to find a replacement of the failed or deleted node for all the affected workloads, without changing the rest of the topology assignment. Currently this works only for a single node failure at the time and in case of multiple failures, the workload gets evicted.

Replace Node on Pod termination

By default, the node is assumed to have failed if it is missing (removed from the cluster), or if its conditions.Status.Ready is not True and the workload’s Pods scheduled on that node are either terminated or terminating. A node with still-running Pods is not considered failed, no matter how long its conditions.Status.Ready has not been True: nodes can recover after an arbitrary amount of time, and reassigning while the Pods are alive makes the stored assignment diverge from where the Pods actually run.

Before v0.19, a node was additionally assumed to have failed once its conditions.Status.Ready was not True for at least 30 seconds, regardless of Pod state. This fixed-time marking is deprecated: it remains available behind the TASReplaceNodeDueToNotReadyOverFixedTime feature gate (enabled by default in v0.17–v0.18, disabled by default and deprecated since v0.19) and is planned for removal. Disabling TASReplaceNodeOnPodTermination retains the pre-v0.14 behavior of marking the node after the fixed time regardless of Pod state.

Note that finding a replacement node that meets all the requirements (e.g. the same type of machine placed in the rack that Kueue had previously assigned to the workload) may not always be possible. If a workload is big enough to cover the whole topology domain (e.g. block or rack) it’s inevitable that there will be no replacement within the same domain. Hence, we recommend using FailFast mode described below or WaitForPodsReady and configuring waitForPodsReady.recoveryTimeout, to prevent the workloads from waiting for the replacement indefinitely.

Fast Hot swap

By default, Kueue tries to find a replacement for a failed node until it succeeds or until the workload is evicted (for example, by waitForPodsReady.recoveryTimeout). To prevent Kueue from retrying indefinitely, you can enable the TASFailedNodeReplacementFailFast feature gate. When enabled, Kueue will only attempt to find a replacement node once. If it fails, it will not try again, and the workload will get evicted and requeued.

Replace Node on Node Taints

Nodes are sometimes tainted to take them out of service, for example to drain them for maintenance or to reserve them for a higher-priority workload. When this happens, the Kubernetes core taint eviction controller may delete the Pods of a running TAS workload, but Kueue’s TAS cache still accounts for the workload on that node. As a result the freed capacity is not reusable until the workload is eventually evicted (for example by waitForPodsReady.recoveryTimeout), which can take several minutes.

With the TASReplaceNodeOnNodeTaints feature gate enabled, a scheduling taint (NoExecute or NoSchedule) that the workload does not tolerate can cause Kueue to treat the node as failed and handle it through the same hot swap flow described above, releasing the stale capacity quickly instead of waiting for the workload to time out. When the node is treated as failed depends on the taint effect and whether the workload’s Pods are still running on it, as described below.

Only the NoExecute and NoSchedule taint effects are considered; nodes with other taint effects (such as PreferNoSchedule) are not treated as failed. Whether a taint is tolerated is evaluated against the effective tolerations of the workload’s Pods for that node, including any tolerations added by admission checks.

When the workload does not tolerate the taint, the node becomes a candidate for replacement. With TASReplaceNodeOnPodTermination enabled (the default), Kueue waits until the workload’s Pods are no longer running on the node (they are terminating, terminated, or were never scheduled there) before treating it as failed; with TASReplaceNodeOnPodTermination disabled, the node is treated as failed as soon as the untolerated taint is observed. The two scheduling taint effects differ in how the Pods stop running:

• NoExecute: The Kubernetes taint eviction controller terminates Pods that do not tolerate the taint, so they stop running on their own. If a Pod tolerates the taint with a tolerationSeconds value, the taint eviction controller keeps that Pod for the given duration before evicting it, so the node is treated as failed only after that Pod stops running.
• NoSchedule: A NoSchedule taint does not evict already-running Pods, so under the default TASReplaceNodeOnPodTermination behavior a workload whose Pods keep running on the tainted node is left undisturbed, and Kueue triggers recovery only after those Pods stop running there, for example after they are drained manually.

In all cases, Kueue replaces the failed node when TASFailedNodeReplacement is enabled, or evicts the workload if no replacement is possible.

Skip reassignment for Workloads owned by a single Pod

Node replacement assumes that the workload’s controller re-creates pods within the same Workload (as Job, JobSet or LeaderWorkerSet do), so that the re-created pods can follow the updated topology assignment. A Workload owned by a single Pod — bare pods and, for example, Deployment replicas managed through the pod integration — does not have this property: the Workload is deleted together with its pod, so no future pod can consume a replacement assignment. Replacing a node for such a Workload only makes its stored assignment diverge from the node its still-running pod occupies, which corrupts the per-node capacity accounting: the pod’s real node is treated as free and gets over-admitted, while the replacement node is blocked for other admissions.

With the SkipReassignmentForPodOwnedWorkloads feature gate enabled, Kueue keeps the existing topology assignment of a Workload owned by exactly one Pod instead of computing a replacement. The failed node is still tracked in .status.unhealthyNodes and the field is cleared on the next successful scheduling pass. Recovery happens through the pod lifecycle: when the pod terminates, its owning controller (for example a ReplicaSet) creates a new pod, which arrives as a new Workload and is admitted with a fresh assignment. Workloads with any other owner, including pod groups (which can receive user-created replacement pods into the same Workload), keep the replacement behavior described above.

The gate also covers the eviction requeue path. Without it, a pod-owned Workload evicted by preemption is requeued and re-admitted with a freshly computed assignment while its pod is still draining the termination grace period on the original node — an assignment no pod can consume, blocking the newly assigned node’s capacity for the full grace period. With the gate enabled, evicted pod-owned Workloads get Requeued=False instead: the Workload finishes with its pod, and the owning controller’s replacement pod arrives as a new Workload.

Usage Scenarios

Here are a few scenarios that can happen when both TASReplaceNodeOnPodTermination and TASFailedNodeReplacementFailFast are enabled:

1. Node becomes NotReady, pods are terminated, and a replacement is found:

   • A node running a pod from a TAS workload becomes NotReady.
   • The pods on that node are terminated.
   • With TASReplaceNodeOnPodTermination enabled, Kueue immediately looks for a replacement.
   • A replacement node is available, and Kueue successfully swaps the failed node.
   • The workload continues running on the new node.

2. Node becomes NotReady, pods are terminated, and no replacement is found:

   • A node running a pod from a TAS workload becomes NotReady.
   • The pods on that node are terminated.
   • Kueue immediately looks for a replacement but cannot find one.
   • With TASFailedNodeReplacementFailFast enabled, Kueue will not retry.
   • The workload immediately gets evicted and requeued (doesn’t wait 30s or until waitForPodsReady.recoveryTimeout expires)

3. Node gets deleted

   • Same scenarios apply as in 1. and 2.

4. The Workload requires the whole rack and one of the nodes becomes NotReady:

   • A node running a pod from a TAS workload becomes NotReady.
   • The pods on that node are terminated.
   • Kueue immediately looks for a replacement but since the workload requires the whole rack, it cannot find the replacement.
   • With TASFailedNodeReplacementFailFast enabled, Kueue does not retry the replacement search.
   • The workload immediately gets evicted and requeued (doesn’t wait 30s or until waitForPodsReady.recoveryTimeout expires)

Feature Gate Interaction Matrix

The following table summarizes the behavior based on the combination of the feature gates. If TASFailedNodeReplacement is false, the other gates have no effect.

Feature Gate Legend:

• FNR: TASFailedNodeReplacement
• RNO: TASReplaceNodeOnPodTermination
• FNFF: TASFailedNodeReplacementFailFast

With the deprecated TASReplaceNodeDueToNotReadyOverFixedTime gate enabled, the not-ready trigger depends on TASReplaceNodeOnPodTermination (RNO):

• RNO enabled: the node is treated as failed when its workload Pods are terminating or terminated, or after 30 seconds of NotReady, whichever comes first.
• RNO disabled: the node is treated as failed only after 30 seconds of NotReady (the pre-v0.14 behavior), regardless of Pod state.

With the gate disabled (the default since v0.19), Pod termination is the only not-ready trigger while RNO remains enabled (its default); disabling RNO retains the fixed-time marking. Nodes that are deleted or lack a Ready condition are treated as failed immediately in all configurations. To disable node replacement entirely, use the TASFailedNodeReplacement feature gate instead.

Recommended configuration

We recommend keeping all four feature gates enabled to ensure the fastest feedback loop for workloads affected by node failures or tainted nodes.

Balanced Placement

The balanced placement algorithm provides an alternative to the greedy packing strategies. Instead of iterating over the domains sorted from largest to smallest available space (or based on some other criteria) and trying to pack as many pods/slices as possible to each domain until the request fits, it first finds the optimal set of domains that fit the request and then distributes the pods/slices as evenly as possible across these domains.

Greedy placement strategies (such as BestFit and LeastFreeCapacity) might result in a placement with a small number of pods assigned to the last considered domain. For example 12 pods distributed among domains with capacities (10,10) will be placed (10,2). However, in some applications, a more balanced placement (6,6) would be more efficient. Some examples of such cases would be all-to-all communication procedures (e.g. Allgather) since more balanced placement leads to more efficient cross-domain traffic.

To use this feature, use the kueue.x-k8s.io/podset-preferred-topology annotation on the Job. Kueue TAS makes sure that the minimum number of pods (or slices) placed on any domain one level below the indicated level will be maximied. Also (as a second criterion) the number of domains used on the indicated level will be minimized. However, if the Job would not fit within a single domain one level above the indicated level, Kueue will not perform the balanced placement and will fallback to the standard TAS algorithm.

Multi-Layer Topology

Multi-layer topology allows you to define up to 3 layers of slice topology constraints, enabling fine-grained placement across deep topology hierarchies. This is useful in data centers with multiple topology levels (e.g., datacenter, block, rack) where you need different slice sizes at each level.

With the single-layer approach (kueue.x-k8s.io/podset-slice-required-topology + kueue.x-k8s.io/podset-slice-size), you can only specify one topology level and one slice size. Multi-layer topology extends this by letting you specify constraints at multiple levels simultaneously.

For example, if you have 64 pods and want groups of 32 pods within the same block and groups of 16 pods within the same rack, you can express this with a single annotation:

kueue.x-k8s.io/podset-slice-required-topology-constraints: |
  [
    {"topology": "cloud.provider.com/topology-block", "size": 32},
    {"topology": "cloud.provider.com/topology-rack", "size": 16}
  ]

The constraints are specified as a JSON array ordered from the outermost (coarsest) to the innermost (finest) topology layer. Each inner slice size must evenly divide the outer slice size.

This annotation is mutually exclusive with kueue.x-k8s.io/podset-slice-required-topology and kueue.x-k8s.io/podset-slice-size.

Drawbacks

When enabling the feature Kueue starts to keep track of all Pods and all nodes in the system, which results in larger memory requirements for Kueue. Additionally, Kueue will take longer to schedule the workloads as it needs to take the topology information into account.