Run A Wrapped Custom Workload

This page shows how to use AppWrappers to make Kueue’s scheduling and resource management capabilities available to Workload types that do not have a dedicated Kueue integration. For Workloads that use PodSpecTemplates in their definition, this can provide a significantly easier approach than building a custom integration to enable the use of Kueue with a custom Workload type.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

1. Make sure you are using Kueue v0.11.0 version or newer and AppWrapper v1.0.0 or newer.

2. Follow the steps in Run AppWrappers to learn how to enable and configure the codeflare.dev/appwrapper integration.

Example using LeaderWorkerSets as the Custom Workload

We use LeaderWorkerSets to explain how to run a workload of a custom type inside an AppWrapper.

1. Follow the install instructions for LeaderWorkerSets.

2. Edit the appwrapper-manager-role ClusterRole to add the stanza below to allow the appwrapper controller to manipulate LeaderWorketSets.

- apiGroups:
  - leaderworkerset.x-k8s.io
  resources:
  - leaderworkersets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

3. The AppWrapper containing the LeaderWorkerSet is shown below. In particular, notice how the replicas and path of each element of the podSets array corresponds to a PodSpecTemplate and replica count within template. This gives the AppWrapper controller enough information to enable it to “understand” the wrapped resource and provide Kueue the information it needs to manage it.

apiVersion: workload.codeflare.dev/v1beta2
kind: AppWrapper
metadata:
  name: sample-lws
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  components:
  - podSets:
    - path: "template.spec.leaderWorkerTemplate.leaderTemplate"
      replicas: 2
    - path: "template.spec.leaderWorkerTemplate.workerTemplate"
      replicas: 3
    template:
      apiVersion: leaderworkerset.x-k8s.io/v1
      kind: LeaderWorkerSet
      metadata:
        name: nginx-leaderworkerset
        labels:
          app: nginx
      spec:
        replicas: 2
        leaderWorkerTemplate:
          leaderTemplate:
            spec:
              containers:
              - name: nginx-leader
                image: registry.k8s.io/nginx-slim:0.27
                resources:
                  requests:
                    cpu: "100m"
                ports:
                - containerPort: 80
          size: 3
          workerTemplate:
            spec:
              containers:
              - name: nginx-worker
                image: nginx:1.14.2
                resources:
                  requests:
                    cpu: "200m"
                ports:
                - containerPort: 80