Run a SparkApplication

This page shows how to leverage Kueue’s scheduling and resource management capabilities when running Spark Operator SparkApplication.

This guide is for batch users that have a basic understanding of Kueue. For more information, see Kueue’s overview.

Before you begin

Enable SparkApplication integration in Kueue. You can modify Kueue configurations from installed releases to include sparkoperator.k8s.io/sparkapplication as an allowed workload.

Enable the SparkApplicationIntegration feature gate. Check the installation guide for details on feature gate configuration.

Check administer cluster quotas for details on the initial cluster setup.

Check the Spark Operator installation guide.

Spark Operator definition

a. Queue selection

The target local queue should be specified in the metadata.labels section of the SparkApplication configuration.

metadata:
  labels:
    kueue.x-k8s.io/queue-name: user-queue

b. Optionally set Suspend field in SparkOperation

spec:
  suspend: true

By default, Kueue will set suspend to true via webhook and unsuspend it when the SparkApplication is admitted.

c. Resource requests

Spark, not the Spark Operator, computes the CPU and memory requests of the driver and executor Pods from the SparkApplication configuration. Kueue reproduces this computation to reserve the quota the Pods will actually use:

• CPU: coreRequest if set, otherwise cores, otherwise 1.
• Memory: memory (default 1g) plus memoryOverhead. If memoryOverhead is not set, the overhead is max(memoryOverheadFactor * memory, 384m), where memoryOverheadFactor defaults to 0.1 for Java and Scala applications and to 0.4 for Python and R applications. Executors of Python applications also get spark.executor.pyspark.memory, and executors get spark.memory.offHeap.size when spark.memory.offHeap.enabled is true.

Memory values use the Java format (for example 512m or 2g). The corresponding properties in spec.sparkConf (for example spark.driver.cores or spark.kubernetes.memoryOverheadFactor) are honored with the same precedence as spark-submit applies: typed driver and executor fields take precedence over spec.sparkConf, which takes precedence over spec.memoryOverheadFactor.

Sample SparkApplication

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  type: Scala
  mode: cluster                 # spark-operator supports "cluster" mode only
  sparkVersion: 4.0.0
  image: spark:4.0.0
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
  arguments:
  - "50000"
  driver:
    coreRequest: "1"
    memory: 1g                  # In Java format (e.g. 512m, 2g)
    serviceAccount: spark       # You need to create this service account beforehand,
                                # and the service account should have proper role
                                # ref: https://github.com/kubeflow/spark-operator/blob/master/config/rbac/spark-application-rbac.yaml
  executor:
    instances: 2
    coreRequest: "1"
    memory: 1g                  # In Java format (e.g. 512m, 2g)
    deleteOnTermination: false  # to keep terminated executor pods for demo purpose