Setup garbage-collection of workload

Configure automatic garbage-collection of finished and deactivated Workloads by defining retention policies.

This guide shows you how to enable and configure optional object retention policies in Kueue to automatically delete finished or deactivated Workloads after a specified time. By default, Kueue leaves all Workload objects in cluster indefinitely; with object retention you can free up etcd storage and reduce Kueue’s memory footprint.

Prerequisites

  • A running Kueue installation at v0.12 or newer.
  • The ObjectRetentionPolicies feature-gate enabled in the Kueue controller manager. Check the Installation guide for details on feature gate configuration.

Set up a retention policy

Follow the instructions described here to install a release version by extending the configuration with the following fields:

      objectRetentionPolicies:
        workloads:
          afterFinished: "1h"
          afterDeactivatedByKueue: "1h"

Workload Retention Policy

The retention policy for Workloads is defined in the .objectRetentionPolicies.workloads field. It contains the following optional fields:

  • afterFinished: Duration after which finished Workloads are deleted.
  • afterDeactivatedByKueue: Duration after which any Kueue-deactivated Workloads (such as a Job, JobSet, or other custom workload types) are deleted.

Example

Kueue Configuration

Configure Kueue with a 1m retention policy and enable waitForPodsReady:

  objectRetentionPolicies:
    workloads:
      afterFinished: "1m"
      afterDeactivatedByKueue: "1m"
  waitForPodsReady:
    enable: true
    timeout: 2m
    recoveryTimeout: 1m
    blockAdmission: true
    requeuingStrategy:
      backoffLimitCount: 0

Scenario A: Successfully finished Workload

  1. Submit a simple Workload that should finish normally:
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: successful-cq
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: [cpu]
    flavors:
    - name: objectretention-rf
      resources:
      - name: cpu
        nominalQuota: "2"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: default
  name: successful-lq
spec:
  clusterQueue: successful-cq
---
apiVersion: batch/v1
kind: Job
metadata:
  name: successful-job
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: successful-lq
spec:
  parallelism: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: c
          image: registry.k8s.io/e2e-test-images/agnhost:2.53
          args: ["entrypoint-tester"]
          resources:
            requests:
              cpu: "1"
  1. Watch the status transition to Finished. After ~1 minute, Kueue will automatically delete it:
# ~1m after Finished
kubectl get workloads -n default
# <successful-workload not found>
   
kubectl get jobs -n default
# NAME             STATUS     COMPLETIONS   DURATION   AGE
# successful-job   Complete   1/1                      1m

Scenario B: Evicted Workload via waitForPodsReady

  1. Configure Kueue deployment to have more resources available than the node can provide:
        resources:
          limits:
            cpu: "100" # or any value greater than the node's capacity
  1. Submit a Workload that requests more than is available on the node:
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: limited-cq
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu"]
    flavors:
    - name: objectretention-rf
      resources:
      - name: cpu
        nominalQuota: "100" # more than is available on the node
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: default
  name: limited-lq
spec:
  clusterQueue: limited-cq
---
apiVersion: batch/v1
kind: Job
metadata:
  name: limited-job
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: limited-lq
spec:
  parallelism: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: c
          image: registry.k8s.io/e2e-test-images/agnhost:2.53
          args: ["entrypoint-tester"]
          resources:
            requests:
              cpu: "100" # more than is available on the node
  1. Without all Pods ready, Kueue evicts and deactivates the Workload:
# ~2m after submission
kubectl get workloads -n default
# NAME                    QUEUE    RESERVED IN   ADMITTED   FINISHED   AGE
# limited-workload                               False                 2m
  1. ~1 minute after eviction, the deactivated Workload is garbage-collected:
# ~1m after eviction
kubectl get workloads -n default
# <limited-workload not found>
   
kubectl get jobs -n default
# <limited-job not found>

Notes

  • afterDeactivatedByKueue is the duration to wait after any Kueue-managed Workload (such as a Job, JobSet, or other custom workload types) has been marked as deactivated by Kueue before automatically deleting it. Deletion of deactivated workloads may cascade to objects not created by Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger garbage-collection of dependent resources. If you manually / automatically deactivate the Workload by specifying .spec.active=false, afterDeactivatedByKueue is NOT effective.
  • If a retention duration is mis-configured (invalid duration), the controller will fail to start.
  • Deletion is handled synchronously in the reconciliation loop; in clusters with thousands of expired Workloads it may take time to remove them all on first startup.

Last modified June 5, 2025: docs: add object retention page (#5515) (aba21166)