Observability on Kueue

Setup Prometheus

Wed, 04 Feb 2026 00:00:00 +0000

This page shows how to set up Prometheus to scrape Kueue metrics. For TLS-secured metrics endpoints, see Configure Prometheus with TLS.

The page is intended for a batch administrator.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
Prometheus Operator is installed.

1. Setup

Choose the setup method that matches your Kueue installation.

Common Grafana Queries

Tue, 03 Feb 2026 00:00:00 +0000

This page shows you how to use common PromQL queries to monitor Kueue metrics in Grafana.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
kube-prometheus is installed.
Kueue Prometheus metrics are enabled (see Setup Prometheus).

Quota utilization

Note

The queries in this section require metrics.enableClusterQueueResources: true in the Kueue configuration. See Installation for details.

To monitor the percentage of CPU quota being used in a ClusterQueue:

Monitor actual resource usage

Tue, 24 Feb 2026 00:00:00 +0000

This page shows you how to use Kueue labels assigned to pods to monitor resource usage.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

A Kubernetes cluster is running.
The kubectl command-line tool has communication with your cluster.
Kueue is installed.
The cluster has at least one local queue and cluster queue. Presented commands assume your local queue is named “user-queue” and cluster queue is named “cluster-queue”, if this is not the case adjust them appropriately.

Note

The queries in this page require AssignQueueLabelsForPods Feature Gate, which is enabled by default. If it is not enabled, see Installation for details how to enable it.

`kubectl top` for command line resource usage debugging

Warning

As mentioned on Metrics Server repository site, Metrics Server and provided command line tool - kubectl top - is not meant as a monitoring solution. The tool is convenient for quick troubleshooting but is not a substitute for full-scale monitoring. See the Production resource monitoring section for a more robust setup.

To use kubectl top you need to install metrics-server for you cluster. Follow the Metrics Server Installation
Schedule a couple of jobs that have some actual cpu usage to your local queue:

for i in {1..3}; do
kubectl create -f - <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
 generateName: sample-job-
 namespace: default
 labels:
 kueue.x-k8s.io/queue-name: "user-queue"
spec:
 parallelism: 1
 completions: 1
 template:
 spec:
 containers:
 - name: dummy-job
 image: registry.k8s.io/e2e-test-images/agnhost:2.53
 command: [ "/bin/sh" ]
 args: [ "-c", "end_time=$(($(date +%s) + 60)); while [ $(date +%s) -lt $end_time ]; do :; done"]
 resources:
 requests:
 cpu: "1"

 restartPolicy: Never
EOF
done

Monitor the usage of cpu and memory in nearly real time for this local queue with kubectl top:

watch -n 15 'kubectl top pod --sum -l kueue.x-k8s.io/local-queue-name=user-queue'

You will see a list of pods currently running on the local queue named “user-queue” with their current cpu and memory measurements and the sum of the usage at the bottom. The list will be automatically refreshed every 15 seconds. The outcome should look like this:

Configure Custom Metric Labels

Tue, 10 Mar 2026 00:00:00 +0000

This page shows you how to configure custom metric labels that promote Kubernetes labels or annotations on ClusterQueues, LocalQueues, and Cohorts into additional Prometheus metric label dimensions.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

Observability on Kueue

Setup Prometheus

Before you begin

1. Setup

Common Grafana Queries

Before you begin

Quota utilization

Note

Monitor actual resource usage

Before you begin

Note

kubectl top for command line resource usage debugging

Warning

Configure Custom Metric Labels

Before you begin

`kubectl top` for command line resource usage debugging