~/makariy
ENRU
← All articles

27 min read

Scheduling GPU workloads in Kubernetes: from standard mechanisms to custom solutions

The full spectrum of GPU scheduling in Kubernetes: from nodeSelector and affinity through KubeSchedulerConfiguration to JobSet, Kueue, Volcano, YuniKorn and KAI-Scheduler — with examples and guidance on what to pick when.

Originally published at

Hi! My name is Makariy, and as a Senior SRE at Yandex Cloud I not only helped build Managed Service for Kubernetes, but always enjoyed spending free time looking at what interesting things had shipped for “the cube” lately. Kubernetes, the de facto standard for container orchestration, offers basic mechanisms for managing compute resources. The default Kubernetes scheduler (kube-scheduler), however, was designed around general load-balancing principles and is not specialised for the peculiarities of GPU workloads.

I want to walk through the whole spectrum — from the scheduling mechanisms built into Kubernetes to specialised schedulers such as Volcano, Apache YuniKorn and KAI-Scheduler. I will look at the concrete scenarios where each of these tools earns its keep, and offer recommendations on picking the right one for your GPU workloads.

What you will find in this article:

What is scheduling in K8s?

By scheduling we mean the process of distributing pods across the nodes of a Kubernetes cluster. That process is driven by a component called kube-scheduler. Based on various criteria — resources, policies, labels and so on — it is responsible for choosing suitable nodes for each pod. These mechanisms are especially useful for GPU workloads, because that kind of capacity is always in short supply these days.

Fine-tuning pod placement in Kubernetes

Let us start by working out when the built-in scheduling mechanisms are enough.

Default Scheduling

Kube-scheduler distributes pods across nodes automatically, based on available resources (CPU, RAM, GPU) and other factors.

Use case: when you cannot be bothered to invent anything and have no requirements about which node the pod lands on.

Node Selector

nodeSelector is the simplest way to restrict pod placement to particular nodes. It lets you name the node label the pod must be scheduled onto.

Use case: a company has a cluster with several GPU types (V100, A100, T4). An inference job specifically needs a V100 for optimal performance.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-inference
spec:
  containers:
  - name: inference-container
    image: ai-model:latest
    resources:
      limits:
        nvidia.com/gpu: 1
  nodeSelector:
    gpu-type: "tesla-v100"

❗️ Do not forget to make sure the node actually carries the label — you can set it either manually or automatically.

Node Name

This is a field in the pod spec that, when set, makes every other scheduling mechanism irrelevant. If the named node is unavailable or cannot run the pod (because it is out of resources, for example), the pod stays Pending.

Use case: performance debugging on specific hardware, or guaranteeing that a critical job runs on a node with known characteristics.

apiVersion: v1
kind: Pod
metadata:
  name: direct-placement-gpu
spec:
  nodeName: gpu-node-0012
  containers:
  - name: gpu-workload
    image: nvidia/cuda:11.0-base
    resources:
      limits:
        nvidia.com/gpu: 1

This option suits testing or manual placement control.

💀 But because it is completely inflexible, you can wave goodbye to your application’s availability and its ability to scale.

Affinity/Anti-Affinity

Mechanisms that let you control how pods are distributed across nodes (Node Affinity/Anti-Affinity) or how they relate to other pods (Pod Affinity/Anti-Affinity), based on labels.

Node Affinity/Anti-Affinity

nodeAffinity offers a more flexible approach than nodeSelector, with support for complex node-selection logic.

Use case 1. An ML team strictly requires V100 GPUs, but prefers specific Yandex Cloud instance types (they cost less).

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu-type
            operator: In
            values:
            - v100
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
            - gpu-standard-v2
  containers:
  - name: training-container
    image: deep-learning:latest
    resources:
      limits:
        nvidia.com/gpu: 4

Use case 2. A critical production inference workload that must avoid nodes where experimental or shared GPU jobs are running. Node Affinity here excludes a particular node type in order to isolate the critical work.

nodeAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
    preference:
      matchExpressions:
      - key: workload-type
        operator: NotIn
        values:
        - gpu-shared

Pod Affinity / Anti-Affinity

Pod Affinity controls pod placement relative to other pods, which matters for complex ML pipelines.

Use case: the inference server must land on the same node as the model-cache pod, to minimise model loading latency.

Pod Affinity example

apiVersion: v1
kind: Pod
metadata:
  name: model-server
  labels:
    app: inference
    model: gpt-j
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - model-cache
        topologyKey: kubernetes.io/hostname
  containers:
  - name: inference-server
    image: model-server:latest
    resources:
      limits:
        nvidia.com/gpu: 1

Use case: spreading several inference service replicas across different nodes to improve availability and fault tolerance.

Pod Anti-Affinity example

podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
    podAffinityTerm:
      labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - inference
      topologyKey: kubernetes.io/hostname

Taint & Tolerations

These let you control which pods may run on which nodes. Nodes can be “tainted” so they reject pods that do not carry a matching “toleration”.

This helps reserve nodes for specific workloads, or prevent certain pods from landing on certain nodes. It is also useful for node maintenance and upgrades.

Reserving GPU nodes for critical workloads

Use case: isolating expensive GPU nodes for production ML workloads, keeping non-critical pods off them.

# Taint the node
kubectl taint nodes gpu-node-1 dedicated=ml-prod:NoSchedule

# Pod with a matching toleration
apiVersion: v1
kind: Pod
metadata:
  name: critical-model-training
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "ml-prod"
    effect: "NoSchedule"
  containers:
  - name: training-job
    image: ml-framework:latest
    resources:
      limits:
        nvidia.com/gpu: 8

Dedicating specific GPUs to specific teams

Use case: splitting GPU resources between teams in a large organisation, where the computer vision team gets dedicated GPUs for its own workloads.

# Node for the computer vision team
kubectl taint nodes gpu-node-2 team=computer-vision:NoSchedule

# Only CV team pods can use these nodes
apiVersion: v1
kind: Pod
metadata:
  name: object-detection-training
spec:
  tolerations:
  - key: "team"
    operator: "Equal"
    value: "computer-vision"
    effect: "NoSchedule"
  containers:
  - name: cv-container
    # ...

❗️ A pod can carry several tolerations in order to match several taints on a node. DaemonSets automatically add tolerations for NoSchedule taints so their pods can run on every node.

Pod Topology Spread Constraints

Pod Topology Spread Constraints distribute pods across topology domains.

Use case: distributed model training where pods must be spread across availability zones for failure resilience, while some node-level locality is still allowed for performance.

apiVersion: v1
kind: Pod
metadata:
  name: distributed-training
  labels:
    app: ml-training
spec:
  topologySpreadConstraints:
  - maxSkew: 1 # Maximum pod skew between domains
    topologyKey: topology.kubernetes.io/zone # Topology domain
    whenUnsatisfiable: DoNotSchedule # What to do when the rule cannot be satisfied
    labelSelector: # Applies only to pods carrying this label
      matchLabels:
        app: ml-training
  - maxSkew: 2
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: ml-training
  containers:
  - name: pytorch-training
    image: pytorch:latest
    resources:
      limits:
        nvidia.com/gpu: 2

The whenUnsatisfiable field takes two values, which define what happens when the rule cannot be met:

  • DoNotSchedule: the pod will not be scheduled. It hangs in Pending.

  • ScheduleAnyway: the pod will be scheduled onto any node.

Priority and Preemption

Mechanisms that let you express how important pods are and decide which ones start first. Particularly useful when resources are scarce.

Priority:

  • Every pod is assigned a priority indicating its importance relative to other pods.

  • That priority is defined through a PriorityClass.

Preemption:

  • When the cluster runs out of resources for high-priority pods, Kubernetes evicts (deletes) lower-priority pods to free capacity.

PriorityClass:

  • A Kubernetes object that defines a priority level. Each class has value (the numeric priority), description and globalDefault (whether this PriorityClass applies to all pods by default).

Examples

Creating a PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: production-critical
value: 1000000
globalDefault: false
description: "Critical production ML inference"

❗️ Note that the higher the value, the higher the priority.

Using a PriorityClass:

apiVersion: v1
kind: Pod
metadata:
  name: customer-facing-inference
spec:
  priorityClassName: production-critical
  containers:
  - name: inference-server
    image: inference:latest
    resources:
      limits:
        nvidia.com/gpu: 1

Use case: an ML service serving customers in real time has the highest priority and can preempt training and test pods when GPU capacity runs short. After preemption the lower-priority pods move to Pending, and the higher-priority ones start.

❗️ Not every pod can be preempted — system pods (kube-system), for instance, are usually protected. Kube-scheduler takes priorities into account when choosing nodes for pods. Higher-priority pods get preference when resources are allocated. Quality of Service (Guaranteed, Burstable, BestEffort) affects the preemption process.

Preempted pods sit in Pending for a short while if the cluster has enough capacity, or a long while if it does not. Because of that, their requested resources (CPU, memory and others) are counted when the scheduler evaluates a node.

To use resources efficiently you have to move them around somehow — with the Descheduler, for example.

Descheduler

Descheduler is a tool that periodically analyses pod placement in the cluster and can move pods around to improve resource usage. For GPU workloads it is especially useful for fighting resource fragmentation and keeping expensive GPUs busy.

Core Descheduler strategies for GPU optimisation

  1. RemoveDuplicates — removes duplicate pods from the same node.

  2. LowNodeUtilization — moves pods off underutilised nodes.

  3. HighNodeUtilization — drains overloaded nodes.

  4. RemovePodsViolatingNodeAffinity — fixes node affinity violations.

  5. RemovePodsViolatingTopologySpreadConstraint — rebalances pod distribution.

Example 1: consolidating GPU resources to free up whole nodes

Use case: in a cluster with 10 GPU nodes, pods are spread unevenly — each node uses only 1–2 of its 8 GPUs. The Descheduler consolidates the load, freeing entire nodes for jobs that need many GPUs (training large models, for instance).

apiVersion: "descheduler/v1alpha1"
kind: "DeschedulerPolicy"
strategies:
  "RemoveDuplicates":
    enabled: true
  "LowNodeUtilization":
    enabled: true
    params:
      nodeResourceUtilizationThresholds:
        thresholds:
          "nvidia.com/gpu": 25  # Nodes with GPU utilisation below 25%
          "cpu": 20
          "memory": 20
        targetThresholds:
          "nvidia.com/gpu": 80  # Target GPU utilisation of 80%
          "cpu": 70
          "memory": 70
      numberOfNodes: 2  # At most 2 nodes may stay underutilised
  "RemovePodsViolatingNodeAffinity":
    enabled: true
    params:
      nodeAffinityType:
      - "requiredDuringSchedulingIgnoredDuringExecution"

Example 2: balancing GPU load across availability zones

Use case: critical ML services should be spread evenly across availability zones, but the distribution drifts over time. The Descheduler restores the balance.

apiVersion: "descheduler/v1alpha1"
kind: "DeschedulerPolicy"
strategies:
  "RemovePodsViolatingTopologySpreadConstraint":
    enabled: true
    params:
      includeSoftConstraints: true
  "LowNodeUtilization":
    enabled: true
    params:
      nodeResourceUtilizationThresholds:
        thresholds:
          "nvidia.com/gpu": 30
        targetThresholds:
          "nvidia.com/gpu": 75
      # Only process the ML namespaces
      evictableNamespaces:
        include:
        - "ml-inference"
        - "ml-training"

Example 3: fixing placement violations by GPU type

Use case: the cluster has nodes with T4 GPUs (for inference) and A100 GPUs (for training). Over time pods can end up on suboptimal nodes as the cluster changes. The Descheduler fixes those violations by moving pods onto nodes with the right GPU type.

apiVersion: "descheduler/v1alpha1"
kind: "DeschedulerPolicy"
strategies:
  "RemovePodsViolatingNodeAffinity":
    enabled: true
    params:
      nodeAffinityType:
      - "requiredDuringSchedulingIgnoredDuringExecution"
      - "preferredDuringSchedulingIgnoredDuringExecution"
  "LowNodeUtilization":
    enabled: true
    params:
      nodeResourceUtilizationThresholds:
        thresholds:
          "nvidia.com/gpu": 20
        targetThresholds:
          "nvidia.com/gpu": 70
      # Only process ML workloads
      evictableNamespaces:
        include:
        - "ml-inference"
        - "ml-training"
        - "ml-experiments"
      # Honour pod selectors so placement stays correct
      labelSelector:
        matchExpressions:
        - key: gpu-workload-type
          operator: In
          values:
          - "inference"
          - "training"

Example 4: an advanced configuration with metrics and filters

Use case: a large ML platform with mixed workload types needs the Descheduler tuned around priorities, pod age and specific labels.

apiVersion: "descheduler/v1alpha1"
kind: "DeschedulerPolicy"
strategies:
  "LowNodeUtilization":
    enabled: true
    params:
      nodeResourceUtilizationThresholds:
        thresholds:
          "nvidia.com/gpu": 25
          "cpu": 20
          "memory": 20
        targetThresholds:
          "nvidia.com/gpu": 75
          "cpu": 70
          "memory": 70
      evictableNamespaces:
        include:
        - "ml-training"
        - "ml-experiments"
      nodeFit: true  # Check the pod can actually be rescheduled
      # Filters that keep critical pods out of scope
      thresholdPriority: 10000  # Leave pods with priority above 10000 alone
      thresholdPriorityClassName: "high-priority-inference"
  "RemovePodsHavingTooManyRestarts":
    enabled: true
    params:
      podRestartThreshold: 5  # Evict pods with more than 5 restarts
      includingInitContainers: true
  "PodLifeTime":
    enabled: true
    params:
      maxPodLifeTimeSeconds: 86400  # 24 hours for experimental pods
      podStatusPhases:
      - "Pending"
      - "PodInitializing"
      labelSelector:
        matchLabels:
          workload-type: "experiment"

Monitoring Descheduler effectiveness

To judge how well the Descheduler is working, watch its own metrics and their effect on GPU utilisation. Descheduler exposes built-in Prometheus metrics, including the number of evicted pods and details of the operations it performed.

Details of the available metrics and examples of their use are in the official Descheduler documentation.

Things worth knowing

  • The Descheduler does not create new pods; it only deletes existing ones so they get rescheduled.

  • Pods with a PodDisruptionBudget are conditionally protected, within the disruption budget.

  • Critical system pods (kube-system) are excluded by default.

  • Start with --dry-run=true to test your configuration.

  • The run frequency has to balance optimisation against system stability.

A couple of combined-approach examples

Scenario 1: a production ML platform with resource separation

Use case: a critical production inference service that must have:

  • High priority, so it can preempt other pods.

  • Placement on dedicated nodes with T4 GPUs (optimal for inference).

  • Isolation from training pods, for stable latency.

  • Distribution across availability zones for fault tolerance.

apiVersion: v1
kind: Pod
metadata:
  name: production-inference
  labels:
    app: inference
    environment: production
spec:
  priorityClassName: high-priority-inference
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values:
            - NVIDIA-T4  # Best fit for inference
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - training  # Avoid nodes running training pods
        topologyKey: kubernetes.io/hostname
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "inference"
    effect: "NoSchedule"
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: inference
  containers:
  - name: inference-container
    image: inference-server:latest
    resources:
      limits:
        nvidia.com/gpu: 1

Scenario 2: distributed training of a large model

Use case: distributed training of a large language model, which requires:

  • Powerful A100 GPUs with plenty of memory.

  • Pods of the same job placed on the same nodes where possible, to optimise inter-node communication.

  • A toleration for the “training” taint, so it can use nodes reserved for training.

  • A large resource request (8 GPUs, hugepages, memory).

apiVersion: v1
kind: Pod
metadata:
  name: distributed-llm-training-worker-1
  labels:
    app: distributed-training
    role: worker
    job-id: llm-training-job-123
spec:
  priorityClassName: medium-priority-training
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values:
            - NVIDIA-A100
          - key: nvidia.com/gpu.memory
            operator: Gt
            values:
            - "40000"  # At least 40GB of memory required
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: job-id
              operator: In
              values:
              - llm-training-job-123
          topologyKey: kubernetes.io/hostname
  tolerations:
  - key: "workload"
    operator: "Equal"
    value: "training"
    effect: "NoSchedule"
  containers:
  - name: training-container
    image: pytorch-distributed:latest
    resources:
      limits:
        nvidia.com/gpu: 8
        hugepages-2Mi: 5Gi
        memory: 128Gi

Customising the default scheduler

The default kube-scheduler can be tuned substantially through KubeSchedulerConfiguration to optimise GPU workload placement, without moving to a fully custom solution. The key advantages of this approach:

  1. Minimal infrastructure change. No extra components or CRDs to install.

  2. Flexibility. You can tune various aspects of scheduling for your particular GPU workloads.

  3. Multiple profiles. You can create different schedulers for different kinds of GPU jobs.

  4. Compatibility. Being a standard Kubernetes component guarantees compatibility and support.

Careful scheduler tuning can, in some cases, remove the need for more complex solutions altogether — especially for medium-sized clusters with predictable GPU workloads.

❗️ Changing the default scheduler’s settings requires access to the control plane, which means this is not an option on managed offerings. You can, however, run a second default scheduler in the cluster under a different name and configure it to your needs. Pods choose which scheduler assigns them through the spec.schedulerName field.

Example 1: optimising against GPU fragmentation

Use case: a limited number of GPUs where maximum utilisation matters. The configuration aims for dense packing, minimising GPU fragmentation. Pods land on nodes that already have GPUs in use, until the node’s resources are exhausted.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: gpu-optimized-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesBalancedAllocation
        weight: 2  # Higher weight for balanced placement
      - name: NodeResourcesFit
        weight: 4  # High weight for dense packing
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated  # Dense-packing strategy
        resources:
        - name: nvidia.com/gpu
          weight: 10  # High weight for GPU
        - name: cpu
          weight: 3
        - name: memory
          weight: 1

Example 2: optimising for even GPU load distribution

Use case: an inference service that needs stable latency and predictable performance. The configuration spreads load evenly across all available GPUs, preventing individual nodes from being overloaded. That minimises resource contention and keeps response times stable.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: gpu-balanced-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesBalancedAllocation
        weight: 5  # Strong emphasis on balanced distribution
      - name: NodeResourcesFit
        weight: 2
      - name: TaintToleration
        weight: 1
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: LeastAllocated  # Even-distribution strategy
        resources:
        - name: nvidia.com/gpu
          weight: 8
        - name: cpu
          weight: 2
        - name: memory
          weight: 1

Example 3: special handling for heavy GPU jobs

Use case: training very large models (LLMs, for example) that need significant GPU memory. The configuration prefers nodes with the most available GPU and memory resources, to prevent pods from being killed by out-of-memory conditions.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: large-model-scheduler
  plugins:
    preFilter:
      enabled:
      - name: NodeResourcesFit
      - name: NodePorts
    filter:
      enabled:
      - name: NodeResourcesFit
      - name: NodeName
      - name: NodeUnschedulable
    preScore:
      enabled:
      - name: InterPodAffinity
      - name: NodeAffinity
      - name: NodeResourcesFit
    score:
      enabled:
      - name: NodeResourcesBalancedAllocation
        weight: 1
      - name: NodeResourcesFit
        weight: 10
  pluginConfig:
  - name: NodeResourcesFit
    args:
      # Extra headroom to avoid OOM situations
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 10
        - name: memory
          weight: 5
        - name: cpu
          weight: 3

Example 4: a scheduler aware of GPU topology and NVLink

Use case: high-performance computing where inter-processor data exchange speed is critical. The scheduler prefers nodes whose GPUs are connected over NVLink, and places pods with NUMA topology in mind.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: nvlink-aware-scheduler
  plugins:
    filter:
      enabled:
      - name: NodeResourcesFit
      - name: NodeAffinity
      - name: PodTopologySpread
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 8
      - name: NodeAffinity
        weight: 5
      - name: PodTopologySpread
        weight: 3
      - name: InterPodAffinity
        weight: 2
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 15
        - name: hugepages-2Mi
          weight: 8  # Matters for high-performance computing
        - name: memory
          weight: 5
        - name: cpu
          weight: 3
  - name: NodeAffinity
    args:
      addedAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: nvidia.com/gpu.nvlink
              operator: Exists
            - key: node.kubernetes.io/instance-type
              operator: In
              values:
              - gpu-h100-8x  # Nodes with 8 H100 GPUs and NVLink

Example 5: a multi-profile scheduler for different workload types

Use case: a general-purpose configuration for a cluster with several kinds of GPU workload. Each profile is optimised for a specific use case.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
# Inference profile - fast placement, even distribution
- schedulerName: inference-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 3
      - name: NodeResourcesBalancedAllocation
        weight: 5
      - name: ImageLocality
        weight: 2  # Take image locality into account
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: LeastAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 8
        - name: cpu
          weight: 3
        - name: memory
          weight: 2

# Training profile - dense packing, maximum utilisation
- schedulerName: training-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 8
      - name: NodeResourcesBalancedAllocation
        weight: 2
      - name: PodTopologySpread
        weight: 3
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 12
        - name: memory
          weight: 6
        - name: hugepages-2Mi
          weight: 4
        - name: cpu
          weight: 2

# Experiment profile - flexible placement at low priority
- schedulerName: experiment-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 4
      - name: NodeResourcesBalancedAllocation
        weight: 3
      - name: TaintToleration
        weight: 2
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: LeastAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 6
        - name: cpu
          weight: 3
        - name: memory
          weight: 2

Example 6: a scheduler with GPU sharing and MIG support

Use case: modern A100/H100 GPUs support Multi-Instance GPU (MIG), letting you split one physical GPU into several virtual ones. This scheduler is optimised to use MIG instances efficiently.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: mig-aware-scheduler
  plugins:
    filter:
      enabled:
      - name: NodeResourcesFit
      - name: NodeAffinity
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 10
      - name: NodeResourcesBalancedAllocation
        weight: 3
      - name: NodeAffinity
        weight: 2
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated  # Maximise MIG slice utilisation
        resources:
        - name: nvidia.com/mig-1g.5gb    # MIG 1/7 A100
          weight: 8
        - name: nvidia.com/mig-2g.10gb   # MIG 2/7 A100
          weight: 10
        - name: nvidia.com/mig-3g.20gb   # MIG 3/7 A100
          weight: 12
        - name: nvidia.com/mig-7g.40gb   # MIG 7/7 A100 (full GPU)
          weight: 15
        - name: memory
          weight: 4
        - name: cpu
          weight: 2

Practical recommendations:

  • Start with simple configurations and add complexity gradually.
  • Test new configurations in a dev environment before production.
  • Monitor scheduler metrics to judge effectiveness.
  • Use different schedulers for different workload types.
  • Revisit configurations regularly as GPU usage patterns change.

Scheduling workloads with JobSet and Kueue

JobSet and Kueue are relatively new tools in the Kubernetes ecosystem that offer elegant solutions for orchestrating complex GPU workloads. Unlike full-blown custom schedulers, they focus on specific aspects of batch job management while still integrating with the standard Kubernetes scheduler.

JobSet is a Kubernetes extension for managing groups of related jobs that need to run together.

Key capabilities:

  • Coordinated start and finish for a group of pods.

  • Lifecycle management for the whole set of jobs.

  • Service discovery between pods in the set.

Use case: distributed model training where one master pod and four worker pods must start and finish together. JobSet coordinates all components, which keeps expensive GPUs from sitting idle.

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: distributed-training
spec:
  replicatedJobs:
    - name: master
      replicas: 1
      template:
        spec:
          template:
            spec:
              containers:
              - name: training
                image: ml-training:latest
                resources:
                  limits:
                    nvidia.com/gpu: 1
    - name: worker
      replicas: 4
      template:
        spec:
          template:
            spec:
              containers:
              - name: training
                image: ml-training:latest
                resources:
                  limits:
                    nvidia.com/gpu: 2

Kueue: queue management for resources

Kueue is a queue management system for Kubernetes that optimises the use of expensive resources through quotas and prioritisation.

Key capabilities:

  • Quota management for different GPU types.

  • Resource sharing between teams and projects.

  • Prioritisation of critical workloads.

  • Awareness of different hardware types (resource flavors).

Use case: an organisation with several ML teams sharing a pool of A100 GPUs. Kueue guarantees that no single team monopolises the expensive resources.

# Define the GPU flavours
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: nvidia-a100
spec:
  nodeLabels:
    gpu-type: a100
---
# Create a cluster queue with quotas
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: ml-queue
spec:
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: nvidia-a100
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 16
---
# Queue for the ML team
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: ml-team
  name: training-jobs
spec:
  clusterQueue: ml-queue

Using JobSet and Kueue together

Use case: distributed model training that needs 8 GPUs. Kueue queues the job and allocates resources according to the team’s quota, while JobSet ensures all pods start as a single unit.

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: distributed-training
  namespace: ml-team
  annotations:
    kueue.x-k8s.io/queue-name: training-jobs  # Kueue integration
spec:
  replicatedJobs:
    - name: training-job
      replicas: 4
      template:
        spec:
          template:
            spec:
              containers:
              - name: training
                image: ml-model:latest
                resources:
                  limits:
                    nvidia.com/gpu: 2

When to use which

  • JobSet. When you need a coordinated start for several pods in distributed training, but not a full gang scheduler like Volcano — with the ability to start all the pods in a job simultaneously.

  • Kueue. When you need to manage different teams’ access to limited GPU resources, but do not need a complex hierarchy (as in YuniKorn).

  • JobSet + Kueue. The optimal solution for most organisations that need both job coordination and fair GPU distribution. This combination is simpler to set up and operate than a full-featured custom scheduler.

Custom Scheduling

The standard Kubernetes scheduling tools are sufficient when you have:

  • Simple GPU jobs with minimal requirements.

  • A limited number of GPU workload types.

  • An environment dominated by single pods.

  • Predictable GPU usage patterns.

Let us look at when the various full-featured custom schedulers become worthwhile.

Volcano Scheduler

Volcano provides a framework for high-performance computing (HPC) and machine learning workloads.

Key capabilities of Volcano Scheduler

  • Gang Scheduling. Guarantees that all pods in a job start at the same time.

  • Priority queues. Lets you define priorities for different workload types.

  • Fair resource sharing. Guarantees resource access for different teams.

An example of multi-GPU model training

In this example Volcano guarantees that all four pods (1 parameter server and 3 workers) start simultaneously, which is critical for distributed training.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: distributed-training
spec:
  minAvailable: 4
  schedulerName: volcano
  plugins:
    env: []
    svc: []
  policies:
    - event: PodEvicted
      action: RestartJob
  tasks:
    - replicas: 1
      name: ps
      template:
        spec:
          containers:
            - image: tensorflow/tensorflow:gpu
              name: tensorflow
              resources:
                limits:
                  nvidia.com/gpu: 1
    - replicas: 3
      name: worker
      template:
        spec:
          containers:
            - image: tensorflow/tensorflow:gpu
              name: tensorflow
              resources:
                limits:
                  nvidia.com/gpu: 2

Apache YuniKorn

YuniKorn is a cloud-native scheduler for containerised workloads, aimed at resource orchestration in large clusters.

Key capabilities of Apache YuniKorn

  • Hierarchical queues. A complex queue structure with policy inheritance.

  • Resource reservation. The ability to reserve resources for future jobs.

  • Quota management. Precise control over how teams consume resources.

An example configuration for model training

In this training scenario YuniKorn guarantees fair GPU distribution between different training jobs while respecting per-team quotas. Modern YuniKorn uses standard Kubernetes resources with annotations.

# Training Job with YuniKorn annotations
apiVersion: batch/v1
kind: Job
metadata:
  name: inference-job
  namespace: ml-inference
  annotations:
    # Point at the YuniKorn queue
    yunikorn.apache.org/queue: root.engineering.ml-team
    # Enable gang scheduling so the pods start together
    yunikorn.apache.org/task-group-name: inference-workers
    yunikorn.apache.org/task-groups: |-
      [{
        "name": "inference",
        "minMember": 4,
        "minResource": {
          "nvidia.com/gpu": 4,
          "memory": "32Gi",
          "cpu": "16"
        },
        "nodeSelector": {
          "gpu-type": "tesla-v100"
        },
        "tolerations": [{
          "key": "dedicated",
          "operator": "Equal",
          "value": "ml-training",
          "effect": "NoSchedule"
        }]
      }]
spec:
  parallelism: 4
  completions: 4
  template:
    metadata:
      labels:
        app: inference
        yunikorn.apache.org/task-group-name: inference-workers
    spec:
      schedulerName: yunikorn
      restartPolicy: Never
      containers:
      - name: inference
        image: inference-server:latest
        resources:
          requests:
            nvidia.com/gpu: 1
            memory: "8Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: 1
            memory: "8Gi"
            cpu: "4"

KAI-Scheduler

KAI-Scheduler specialises in AI/ML workloads, with a deep understanding of GPU topology and the requirements of training pipelines.

Key capabilities of KAI-Scheduler

  • Topology optimisation. Places pods with NVLink and other inter-GPU interconnects in mind.

  • GPU usage prediction. Analyses usage patterns for optimal scheduling.

  • SLA-based prioritisation. Takes SLAs for critical ML jobs into account.

An example for training a large model

Here KAI-Scheduler places a large language model training job on a node with 8 GPUs connected over NVLink, giving maximum inter-processor data exchange performance.

apiVersion: scheduling.kai.io/v1
kind: GPUTask
metadata:
  name: llm-training
spec:
  schedulerName: kai-scheduler
  priority: high
  topologyAware: true
  gpuCount: 8
  nvlinkRequired: true
  template:
    spec:
      containers:
      - name: training
        image: llm-training:latest
        resources:
          limits:
            nvidia.com/gpu: 8
            memory: "128Gi"

Edge cases and recommendations

When the default scheduler is the right call

  1. A small cluster (under 50 GPUs) with predictable load.

  2. Homogeneous GPU jobs with no special inter-node communication requirements.

  3. Clear resource separation between teams, based on dedicated node pools.

  4. No complex pipelines with many dependencies.

  5. No hard requirements around starting groups of pods simultaneously.

When it is time to move to a custom scheduler

  1. JobSet + Kueue:

    • You need a coordinated start for groups of pods in distributed training, but a full gang scheduler is overkill.
    • You need fair GPU distribution between teams without a complex hierarchy.
    • You want a queue management system with quotas that is simple to configure.
    • Integration with the standard Kubernetes scheduler matters, without replacing its core functionality.

  2. Volcano:

    • Distributed training runs are frequently blocked by lack of resources.
    • Complaints about unfair GPU distribution between teams are appearing.
    • You need to start many dependent pods as a single job.

  3. YuniKorn:

    • The infrastructure is scaling to hundreds of GPUs with dozens of teams.
    • You need to reallocate quotas between departments dynamically.
    • A complex hierarchy of resource access is becoming necessary.

  4. KAI-Scheduler:

    • Performance depends critically on specific GPU characteristics.
    • A significant investment in expensive GPUs (A100/H100) demands maximum efficiency.
    • You have complex heterogeneous workloads with different optimal configurations.

Practical migration examples

Example 1: from the default scheduler to Volcano

Starting point. An ML research team uses the standard kube-scheduler:

apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training
spec:
  parallelism: 4
  template:
    spec:
      containers:
      - name: training
        image: training:latest
        resources:
          limits:
            nvidia.com/gpu: 1

Problem. Because GPUs are scarce, only some of the pods start; the rest stay Pending, blocking the already-running pods that are waiting for them.

Solution with Volcano:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: distributed-training
spec:
  minAvailable: 4
  schedulerName: volcano
  tasks:
    - replicas: 4
      name: worker
      template:
        spec:
          containers:
          - name: training
            image: training:latest
            resources:
              limits:
                nvidia.com/gpu: 1

Result. Either all pods start at once or they all wait together, which removes the resource-blocking problem and raises overall GPU utilisation in the cluster.

Example 2: from the default scheduler to YuniKorn

Starting point. A company splits resources manually, using separate namespaces with ResourceQuotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    nvidia.com/gpu: 8
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-b-quota
  namespace: team-b
spec:
  hard:
    nvidia.com/gpu: 8

Problem. Team A frequently does not use all its allocated GPUs, while team B is starved and its pods sit in long queues.

Solution with YuniKorn:

# YuniKorn configuration
partitions:
  - name: default
    queues:
      - name: root
        queues:
          - name: team-a
            resources:
              guaranteed:
                nvidia.com/gpu: 4
              max:
                nvidia.com/gpu: 12
          - name: team-b
            resources:
              guaranteed:
                nvidia.com/gpu: 4
              max:
                nvidia.com/gpu: 12

Result. Each team gets a guaranteed GPU minimum but can use more when capacity is free. YuniKorn dynamically hands team A’s unused GPUs to team B, raising overall resource utilisation by 40%.

Example 3: from Volcano to KAI-Scheduler

Starting point. A research lab uses Volcano to train large language models:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: llm-training
spec:
  minAvailable: 8
  schedulerName: volcano
  tasks:
    - replicas: 8
      name: training
      template:
        spec:
          containers:
          - name: llm-container
            image: llm-training:latest
            resources:
              limits:
                nvidia.com/gpu: 1

Problem. Training runs inefficiently because pods are placed at random and can end up on different nodes with limited inter-node network bandwidth.

Solution with KAI-Scheduler:

apiVersion: scheduling.kai.io/v1
kind: GPUWorkload
metadata:
  name: llm-training
spec:
  schedulerName: kai-scheduler
  topologyAware: true
  preferSameNode: true
  communicationPattern: allToAll
  template:
    spec:
      containers:
      - name: llm-container
        image: llm-training:latest
        resources:
          limits:
            nvidia.com/gpu: 8

Result. KAI-Scheduler places all 8 GPUs for the job on a single node with NVLink interconnects, cutting training time by 35% by removing the inter-node communication bottleneck.

Conclusions and final recommendations

  1. Start simple. For most ML workloads, the default scheduler optimised through KubeSchedulerConfiguration — with the right Node Affinity, Priority Class and Pod Topology Spread Constraints settings — may well be enough.

  2. Consider JobSet and Kueue as an intermediate step. When the default scheduler is not enough but a full custom solution is overkill, JobSet (for coordinating groups of pods) combined with Kueue (for queue management) gives you an effective solution with minimal overhead.

  3. Monitor the bottlenecks. Track GPU utilisation metrics, queue wait times and resource-blocking problems to work out whether you actually need a custom scheduler.

  4. Pick the scheduler by your main problem:

    • Volcano — when the main problem is gang scheduling and basic queue management.
    • YuniKorn — when you need complex hierarchical resource management at enterprise scale.
    • KAI-Scheduler — when performance optimisation at the level of GPU hardware topology is critical.

  5. Take an evolutionary approach:

    • Start by optimising the default scheduler through KubeSchedulerConfiguration.
    • Add JobSet for job coordination and/or Kueue for queue management if you need to.
    • Move to a full custom scheduler only when there is a justified need.

  6. Start with a pilot. Roll a custom scheduler out to a limited set of workloads first, then expand gradually.

  7. Combine when it makes sense. In very complex environments you can run several schedulers for different workload types — a customised default scheduler with Kueue for inference, say, and Volcano for distributed training.

Any change to the scheduling system should be justified by real needs and measurable improvements in resource efficiency, performance or infrastructure manageability.

For most organisations, the best approach is to add complexity gradually: start by tuning the standard tools, and move to specialised solutions only when you genuinely need them.

Further reading: