Configure Custom Metric Labels

This page shows you how to configure custom metric labels that promote Kubernetes labels or annotations on ClusterQueues, LocalQueues, Cohorts, and Workloads into additional Prometheus metric label dimensions.

The intended audience for this page are batch administrators.

Before you begin

Make sure the following conditions are met:

• A Kubernetes cluster is running.
• The kubectl command-line tool has communication with your cluster.
• Kueue is installed.
• Prometheus is set up to scrape Kueue metrics.

Configure custom labels

Custom metric labels require the CustomMetricLabels feature gate, which is alpha and disabled by default. See Installation for details on how to enable feature gates.

Add a metrics.customLabels section to the Kueue controller configuration. Each entry defines one extra Prometheus label dimension. The sourceKind field selects the object the value is read from: ClusterQueue (the default), LocalQueue, Cohort, or Workload.

apiVersion: config.kueue.x-k8s.io/v1beta2
kind: Configuration
featureGates:
  CustomMetricLabels: true
metrics:
  customLabels:
    - name: team
    - name: cost_center
      sourceLabelKey: "billing/cost-center"
    - name: budget
      sourceAnnotationKey: "billing.example.com/budget"
    - name: workload_type
      sourceKind: Workload
      sourceLabelKey: "company.com/workload-type"
      trackedValues: ["inference", "training"]

Field reference

The number of custom labels is limited to keep metric cardinality bounded: up to 2 labels with sourceKind: Workload, up to 6 for each other kind, and 20 in total. The configuration is validated at startup, so an invalid entry prevents the controller from starting.

How values are resolved

For each object, Kueue reads the value of the configured Kubernetes label or annotation. If the object does not have it, an empty string is used.

For ClusterQueue, LocalQueue, and Cohort labels the value is read from the object itself. For Workload labels the value is read from the Workload; Kueue copies the configured source key from the underlying job (Job, JobSet, and other supported kinds) onto the Workload it creates, so you set the label or annotation on the job.

A Workload label must declare its allowed values in trackedValues. Workloads are numerous and can carry arbitrary values, so any value outside the list is reported under kueue.x-k8s.io/_UNTRACKED_VALUE_. This keeps the number of time series bounded regardless of how many distinct values workloads use.

Example: label by team

Using the configuration above with name: team:

1. Create a ClusterQueue with a team label:

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ClusterQueue
   metadata:
     name: engineering-cq
     labels:
       team: platform
   spec:
     resourceGroups:
       - coveredResources: ["cpu"]
         flavors:
           - name: default
             resources:
               - name: cpu
                 nominalQuota: 10
   

2. Query metrics filtered by team:

   kueue_pending_workloads{custom_team="platform"}
   kueue_admitted_workloads_total{custom_team="platform"}
   

Example: break down workloads by type

Workload labels split a workload count metric by a label or annotation carried by individual workloads instead of by a queue. Currently they apply to the kueue_admitted_active_workloads metric.

Using the workload_type entry from the configuration above:

1. Submit jobs carrying the company.com/workload-type label to a ClusterQueue cq-1 labeled team=platform.

2. kueue_admitted_active_workloads carries both custom_team (from the ClusterQueue) and custom_workload_type (from each workload). With 10 active inference workloads, 20 training, and 3 with some other value:

   kueue_admitted_active_workloads{cluster_queue="cq-1", ..., custom_team="platform", custom_workload_type="inference"} 10
   kueue_admitted_active_workloads{cluster_queue="cq-1", ..., custom_team="platform", custom_workload_type="training"} 20
   kueue_admitted_active_workloads{cluster_queue="cq-1", ..., custom_team="platform", custom_workload_type="kueue.x-k8s.io/_UNTRACKED_VALUE_"} 3
   

3. Aggregate by workload type:

   sum by (custom_workload_type) (kueue_admitted_active_workloads)
   

What’s next

• See Prometheus Metrics for the full list of Kueue metrics.
• See Common Grafana Queries for PromQL queries to monitor Kueue in Grafana.