Run KubeRay Jobs in Multi-Cluster
Before you begin
Check the MultiKueue installation guide on how to properly setup MultiKueue clusters.
For the ease of setup and use we recommend using at least Kueue v0.11.0 and for KubeRay Operator at least v1.3.1.
See KubeRay Operator Installation for installation and configuration details of KubeRay Operator.
Note
Before the ManagedBy feature was supported in Kueue (below v0.11.0), the installation of KubeRay Operator in the Manager Cluster must be limited to CRDs only.
To install the CRDs run:
kubectl create -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.3.0"
MultiKueue integration
Once the setup is complete you can test it by running a RayJob ray-job-sample.yaml.
Note
Note: Kueue defaults the spec.managedBy field to kueue.x-k8s.io/multikueue on the management cluster for KubeRay Jobs (RayJob, RayCluster, RayService).
This allows the KubeRay Operator to ignore the Jobs managed by MultiKueue on the management cluster, and in particular skip Pod creation.
The pods are created and the actual computation will happen on the mirror copy of the Job on the selected worker cluster. The mirror copy of the Job does not have the field set.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.