Pending Workloads on-demand
Monitor pending Workloads with the on-demand visibility API
This page shows you how to monitor pending workloads with VisibilityOnDemand feature.
The intended audience for this page are batch administrators, and batch users for Local Queue Visibility section.
From version v0.6.0, Kueue provides the ability for a batch administrators to monitor the pipeline of pending jobs, and help users to estimate when their jobs will start.
Before you begin
Make sure the following conditions are met:
- A Kubernetes cluster is running.
- The kubectl command-line tool has communication with your cluster.
- Kueue is installed in version v0.6.0 or later.
Enable the VisibilityOnDemand feature gate
VisibilityOnDemand is an Alpha feature disabled by default. To use the visibility API change the feature gates configuration and set VisibilityOnDemand=true.
Install the visibility API
To install the visibility API, run the following command
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/visibility-api.yaml
Monitor pending workloads on demand
Feature state stable since Kueue v0.6
To install a simple setup of ClusterQueue
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
name: "default-flavor"
spec:
nodeLabels:
key1: value
key2: value
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: "cluster-queue"
spec:
namespaceSelector: {} # match all.
resourceGroups:
- coveredResources: ["cpu", "memory"]
flavors:
- name: "default-flavor"
resources:
- name: "cpu"
nominalQuota: 9
- name: "memory"
nominalQuota: 36Gi
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
namespace: "default"
name: "user-queue"
spec:
clusterQueue: "cluster-queue"
run the following command:
kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml
Now, let’s create 6 jobs
apiVersion: batch/v1
kind: Job
metadata:
generateName: sample-job-
namespace: default
labels:
kueue.x-k8s.io/queue-name: user-queue
spec:
parallelism: 3
completions: 3
suspend: true
template:
spec:
containers:
- name: dummy-job
image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
args: ["30s"]
resources:
requests:
cpu: 1
memory: "200Mi"
restartPolicy: Never
using a command:
for i in {1..6}; do kubectl create -f https://kueue.sigs.k8s.io/examples/jobs/sample-job.yaml; done
3 of them saturate the ClusterQueue and the other 3 should be pending.
Cluster Queue visibility
To view pending workloads in ClusterQueue cluster-queue run the following command:
kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1alpha1/clusterqueues/cluster-queue/pendingworkloads"
You should get results similar to:
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jrjfr-8d56e",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jrjfr",
"uid": "5863cf0e-b0e7-43bf-a445-f41fa1abedfa"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 0,
"positionInLocalQueue": 0
},
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
},
{
"metadata": {
"name": "job-sample-job-t9b8m-4e770",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-t9b8m",
"uid": "64c26c73-6334-4d13-a1a8-38d99196baa5"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 2,
"positionInLocalQueue": 2
}
]
}
You can pass optional query parameters:
- limit
<integer>- 1000 on default. It indicates max number of pending workloads that should be fetched. - offset
<integer>- 0 by default. It indicates position of the first pending workload that should be fetched, starting from 0.
To view only 1 pending workloads use, starting from position 1 in ClusterQueue run:
kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1alpha1/clusterqueues/cluster-queue/pendingworkloads?limit=1&offset=1"
You should get results similar to
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
}
]
}
Local Queue visibility
Similarly to ClusterQueue, to view pending workloads in LocalQueue user-queue run the following command:
kubectl get --raw /apis/visibility.kueue.x-k8s.io/v1alpha1/namespaces/default/localqueues/user-queue/pendingworkloads
You should get results similar to:
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jrjfr-8d56e",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jrjfr",
"uid": "5863cf0e-b0e7-43bf-a445-f41fa1abedfa"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 0,
"positionInLocalQueue": 0
},
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
},
{
"metadata": {
"name": "job-sample-job-t9b8m-4e770",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-t9b8m",
"uid": "64c26c73-6334-4d13-a1a8-38d99196baa5"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 2,
"positionInLocalQueue": 2
}
]
}
You can pass optional query parameters:
- limit
<integer>- 1000 on default. It indicates max number of pending workloads that should be fetched. - offset
<integer>- 0 by default. It indicates position of the first pending workload that should be fetched, starting from 0.
To view only 1 pending workloads use, starting from position 1 in LocalQueue run:
kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1alpha1/localqueues/user-queue/pendingworkloads?limit=1&offset=1"
You should get results similar to
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
}
]
}
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.