Run A Plain Pod

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running plain Pods.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

1. By default, the integration for v1/pod is not enabled. Learn how to install Kueue with a custom manager configuration and enable the pod integration.

   A configuration for Kueue with enabled pod integration would look like follows:

   apiVersion: config.kueue.x-k8s.io/v1beta1
   kind: Configuration
   integrations:
     frameworks:
      - "pod"
     podOptions:
       # You can change namespaceSelector to define in which 
       # namespaces kueue will manage the pods.
       namespaceSelector:
         matchExpressions:
         - key: kubernetes.io/metadata.name
           operator: NotIn
           values: [ kube-system, kueue-system ]
       # Kueue uses podSelector to manage pods with particular 
       # labels. The default podSelector will match all the pods. 
       podSelector:
         matchExpressions:
         - key: kueue-job
           operator: In
           values: [ "true", "True", "yes" ]
   

2. Kueue will run webhooks for all created pods if the pod integration is enabled. The webhook namespaceSelector could be used to filter the pods to reconcile. The default webhook namespaceSelector is:

   matchExpressions:
   - key: kubernetes.io/metadata.name
     operator: NotIn
     values: [ kube-system, kueue-system ]
   

   When you install Kueue via Helm, the webhook namespace selector will match the integrations.podOptions.namespaceSelector in the values.yaml.

   Make sure that namespaceSelector never matches the kueue namespace, otherwise the Kueue deployment won’t be able to create Pods.

3. Pods that belong to other API resources managed by Kueue are excluded from being queued by pod integration. For example, pods managed by batch/v1.Job won’t be managed by pod integration.

4. Check Administer cluster quotas for details on the initial Kueue setup.

Pod definition

When running Pods on Kueue, take into consideration the following aspects:

a. Queue selection

The target local queue should be specified in the metadata.labels section of the Pod configuration.

metadata:
  labels:
    kueue.x-k8s.io/queue-name: user-queue

b. Configure the resource needs

The resource needs of the workload can be configured in the spec.containers.

    - resources:
        requests:
          cpu: 3

c. The “managed” label

Kueue will inject the kueue.x-k8s.io/managed=true label to indicate which pods are managed by it.

d. Limitations

• A Kueue managed Pod cannot be created in kube-system or kueue-system namespaces.
• In case of preemption, the Pod will be terminated and deleted.

Example Pod

Here is a sample Pod that just sleeps for a few seconds:

apiVersion: v1
kind: Pod
metadata:
  generateName: kueue-sleep-
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  containers:
    - name: sleep
      image: busybox
      command:
        - sleep
      args:
        - 3s
      resources:
        requests:
          cpu: 3
  restartPolicy: OnFailure

You can create the Pod using the following command:

# Create the pod
kubectl apply -f kueue-pod.yaml