Setup Topology-Aware Scheduling

A hands-on guide to configuring Topology-Aware Scheduling and observing how it places workloads.

This page shows how to set up Topology-Aware Scheduling (TAS) as a cluster administrator, and how to observe the decisions Kueue makes. We use a local Kind cluster so that you can follow the steps on your machine, but the same steps apply to any cluster whose nodes carry topology labels.

The intended audience for this page are batch administrators. If you are a batch user looking to run workloads on an already configured cluster, see Run Workloads with Topology-Aware Scheduling.

Before you begin

Make sure the following conditions are met:

The kubectl command-line tool is installed.
The jq command-line JSON processor is installed.
Kind is installed.
The TopologyAwareScheduling feature gate is enabled (beta, on by default).

Step 1: A cluster with topology labels

TAS relies on node labels that describe the placement of each node in the physical hierarchy of your data center, for example its block, rack, and hostname. On cloud providers these labels are typically set by the provider (for details and cloud-specific examples, see Node topology information and Ecosystem Resources); on-premise, you have to label the nodes yourself.

For this guide, create a Kind cluster with 8 worker “nodes” spread over 2 blocks and 4 racks, simulated with labels:

---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiVersion: kubeadm.k8s.io/v1beta3
        scheduler:
          extraArgs:
            v: "3"
        controllerManager:
          extraArgs:
            v: "3"
        apiServer:
          extraArgs:
            enable-aggregator-routing: "true"
            v: "3"
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            v: "3"
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b1
      cloud.provider.com/topology-rack: r1
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b1
      cloud.provider.com/topology-rack: r1
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b1
      cloud.provider.com/topology-rack: r2
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b1
      cloud.provider.com/topology-rack: r2
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b2
      cloud.provider.com/topology-rack: r1
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b2
      cloud.provider.com/topology-rack: r1
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b2
      cloud.provider.com/topology-rack: r2
  - role: worker
    labels:
      cloud.provider.com/node-group: tas-group
      cloud.provider.com/topology-block: b2
      cloud.provider.com/topology-rack: r2

kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        v: "3"

kind create cluster --config kind-cluster.yaml

Then install Kueue.

Verify the topology labels on the nodes:

kubectl get nodes -L cloud.provider.com/topology-block,cloud.provider.com/topology-rack

Step 2: Create the Topology, ResourceFlavor and queues

The Topology object defines the hierarchy of the labels, from the widest (block) to the narrowest (hostname). The ResourceFlavor references it via topologyName, and selects the TAS nodes via nodeLabels:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: "default"
spec:
  levels:
  - nodeLabel: "cloud.provider.com/topology-block"
  - nodeLabel: "cloud.provider.com/topology-rack"
  - nodeLabel: "kubernetes.io/hostname"
---
kind: ResourceFlavor
apiVersion: kueue.x-k8s.io/v1beta2
metadata:
  name: "tas-flavor"
spec:
  nodeLabels:
    cloud.provider.com/node-group: "tas-group"
  topologyName: "default"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "tas-cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "tas-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 100
      - name: "memory"
        nominalQuota: 100Gi
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "tas-user-queue"
spec:
  clusterQueue: "tas-cluster-queue"

kubectl apply -f https://kueue.sigs.k8s.io/examples/tas/sample-queues.yaml

Note

In sample-queues.yaml, the nominalQuota is intentionally set high (100 CPUs) so that ClusterQueue quota is not a limiting factor. This guarantees that scheduling rejections during topology experiments are driven strictly by Kueue’s Topology-Aware Scheduling (TAS) placement constraints rather than quota exhaustion.

Step 3: Run a workload with a topology constraint

Submit a Job that requests all its Pods to run within a single rack, using the kueue.x-k8s.io/podset-required-topology annotation on the Pod template:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: tas-sample-required
  labels:
    kueue.x-k8s.io/queue-name: tas-user-queue
spec:
  parallelism: 10
  completions: 10
  completionMode: Indexed
  template:
    metadata:
      annotations:
        kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-rack"
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
      restartPolicy: Never

kubectl create -f https://kueue.sigs.k8s.io/examples/tas/sample-job-required.yaml

Step 4: Observe the placement decision

Kueue records the chosen topology domains in the Workload’s admission status. Inspect it:

kubectl get workloads
WORKLOAD=$(kubectl get workload -o name --sort-by='.metadata.creationTimestamp' | tail -n 1)
kubectl get $WORKLOAD -o jsonpath='{.status.admission.podSetAssignments[0].topologyAssignment}' | jq

The output enumerates the exact nodes assigned to the Pods, with a Pod count per node, for example:

{
  "levels": [
    "kubernetes.io/hostname"
  ],
  "slices": [
    {
      "domainCount": 2,
      "podCounts": {
        "individual": [
          7,
          3
        ]
      },
      "valuesPerLevel": [
        {
          "individual": {
            "prefix": "kind-worker",
            "roots": [
              "",
              "2"
            ]
          }
        }
      ]
    }
  ]
}

Note

Kueue uses prefix compression in valuesPerLevel status output. In the example above, prefix: "kind-worker" paired with roots: ["", "2"] represents the node names kind-worker (kind-worker + "") and kind-worker2 (kind-worker + "2"). The exact pod distribution in this example assumes worker nodes with 8 allocatable CPUs each; if your local environment allocates a different CPU capacity per node, Kueue will adjust the pod counts across nodes while strictly enforcing the single-rack topology constraint.

Confirm that the Pods landed on the assigned nodes, and that those nodes are in the same rack:

kubectl get pods -o wide
kubectl get nodes -L cloud.provider.com/topology-block,cloud.provider.com/topology-rack

The Pods of a TAS workload are created with the kueue.x-k8s.io/topology scheduling gate, and stay gated until the assignment is computed. Kueue then enforces the assignment by injecting a node selector matching the assigned topology domains into each Pod, and removes the gate.

Step 5: Experiment with the constraint types

To build intuition, try the other annotation variants in sequence and observe how Kueue handles them:

Preferred topology: multi-domain fallback

The kueue.x-k8s.io/podset-preferred-topology annotation tells Kueue that a topology level is preferred, but allows falling back to wider levels or distributing Pods across multiple topology domains if a single domain is full:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: tas-sample-preferred
  labels:
    kueue.x-k8s.io/queue-name: tas-user-queue
spec:
  parallelism: 40
  completions: 40
  completionMode: Indexed
  template:
    metadata:
      annotations:
        kueue.x-k8s.io/podset-preferred-topology: "cloud.provider.com/topology-block"
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
      restartPolicy: Never

Submit the preferred Job while the initial required Job is running:

kubectl create -f https://kueue.sigs.k8s.io/examples/tas/sample-job-preferred.yaml

Observe that Kueue admits the Job and distributes its 40 Pods across the remaining available racks and blocks, consuming the remaining topology capacity across the cluster.

Required topology: admission control by topology placement (under sufficient quota)

Submit an additional required Job that requires all Pods to fit within a single rack:

kubectl create -f https://kueue.sigs.k8s.io/examples/tas/sample-job-required.yaml

Because existing workloads occupy capacity across the racks, no single rack has 10 CPUs available to fit all Pods together. Even though ClusterQueue quota is sufficient (50 CPUs used out of 100 CPUs configured), Kueue’s TAS engine controls resource admission directly by holding the Job in Pending state.

Query the admission failure condition message:

PENDING_WORKLOAD=$(kubectl get workload -o name --sort-by='.metadata.creationTimestamp' | tail -n 1)
kubectl get $PENDING_WORKLOAD -o jsonpath='{.status.conditions[?(@.type=="QuotaReserved")].message}'

The output is similar to:

couldn't assign flavors to pod set main: topology "default" allows to fit only 4 out of 10 pod(s). Total nodes: 8; excluded: resource "cpu": 6

This message proves that TAS actively enforces topology domain placement rules during admission, holding the workload in queue even when ample cluster quota remains.

Unconstrained topology: fill scattered gaps & consume remaining capacity

The kueue.x-k8s.io/podset-unconstrained-topology annotation tells Kueue to schedule Pods on any available nodes without topology domain boundaries, while still using TAS bookkeeping to pack Pods into existing partially-filled nodes:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: tas-sample-unconstrained
  labels:
    kueue.x-k8s.io/queue-name: tas-user-queue
spec:
  parallelism: 6
  completions: 6
  completionMode: Indexed
  template:
    metadata:
      annotations:
        kueue.x-k8s.io/podset-unconstrained-topology: "true"
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["pause"]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
      restartPolicy: Never

Submit the unconstrained Job to fill the remaining node gaps:

kubectl create -f https://kueue.sigs.k8s.io/examples/tas/sample-job-unconstrained.yaml

Observe that Kueue admits the Job, gathering the remaining free CPU slots across scattered nodes in the cluster to consume the remaining cluster capacity:

WORKLOAD=$(kubectl get workload -o name --sort-by='.metadata.creationTimestamp' | tail -n 1)
kubectl get $WORKLOAD -o jsonpath='{.status.admission.podSetAssignments[0].topologyAssignment}' | jq

The output confirms that Pods were packed into the remaining non-full nodes across multiple domains, for example:

{
  "levels": [
    "kubernetes.io/hostname"
  ],
  "slices": [
    {
      "domainCount": 2,
      "podCounts": {
        "individual": [
          4,
          2
        ]
      },
      "valuesPerLevel": [
        {
          "individual": {
            "prefix": "kind-worker",
            "roots": [
              "2",
              "8"
            ]
          }
        }
      ]
    }
  ]
}

Cleanup

kind delete cluster

What’s next

Learn about all PodSet annotations available to batch users.
Explore advanced features like Node Hotswap and failure recovery in Topology-Aware Scheduling Concepts.
Combine TAS with MultiKueue.
Read about TAS drawbacks.

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified July 23, 2026: [website] docs: add hands-on guide for Topology Aware Scheduling setup (#13407) (830031377)