Queues and quotas on Kubernetes with Kueue

Goal

Kubernetes starts every pod it can place, as soon as it can place it. A shared cluster usually wants something in front of that: a queue that decides which team’s work starts, within a quota, and what happens when two teams want the same capacity. Slurm gave me that in Part 1 with its scheduler and accounting. Kueue is a Kubernetes-native job queue that does a similar job for batch work.

I wanted to run two teams against one small cluster, with a quota each, and see how borrowing, reclaiming and all-or-nothing admission behave. This is the first post in Part 5, a platform layer on top of the cluster from Parts 2 and 3. The lab has three small nodes and no GPUs, so the numbers describe that.

Terms used in this post

These build on the earlier glossaries.

  • Job queue: a place where work waits until it is allowed to start.
  • Quota: the amount of CPU and memory a team is entitled to run at once.
  • Admission: Kueue’s decision that a job fits within its team’s quota and may start. It is separate from scheduling, which places pods on nodes.
  • ClusterQueue: a pool of quota, in Kueue.
  • LocalQueue: a queue inside a namespace that points to a ClusterQueue. Teams submit to it.
  • Cohort: a group of ClusterQueues that can borrow each other’s unused quota.
  • Borrowing and reclaiming: using another queue’s idle quota, and taking it back when its owner submits work.
  • Preemption: stopping a running job so that others can use its resources.
  • Gang scheduling: starting all the pods of a job together or not at all.

What I built

Kueue 0.20.4 installed from its release manifest. It runs as one controller pod that asks for 500m of CPU and 512 Mi of memory, so I first scaled the Ray workers from the previous post to zero to make room, because the nodes already had 81% to 91% of their CPU requested.

The setup is small:

  • Two namespaces, team-a and team-b, each with a LocalQueue called main.
  • Two ClusterQueues, one per team, each with a quota of 2 CPUs and 1 GiB of memory, in one cohort called lab, so the cohort has 4 CPUs.
  • Both queues are set to take their quota back from borrowers.
  • Test jobs are busybox containers that sleep, each asking for 1 CPU and 128 Mi, with a label that sends them to the team’s queue.

Two queues in one cohort Diagram: each team’s jobs go through a local queue to a cluster queue, and the two cluster queues share a cohort.

How it works

A job with the queue label is created suspended. Kueue creates a workload for it and compares its requests with the queue’s quota, plus whatever it may borrow from the cohort. If it fits, Kueue admits the workload and unsuspends the job, and only then does the Kubernetes scheduler place its pods. If it does not fit, the job stays suspended and no pods exist for it. When the owner of a borrowed quota submits work, Kueue can preempt the borrower’s workloads to take the quota back.

Where a node provisioner such as Karpenter fits

Kueue decides whether a workload may start, based on quota. It does not add machines. On a cloud cluster something else does, and the usual choices are the Kubernetes Cluster Autoscaler and Karpenter. Karpenter’s overview says it watches for pods the Kubernetes scheduler has marked unschedulable, evaluates their scheduling constraints, provisions nodes that meet them, and disrupts the nodes when they are no longer needed. The two tools therefore act in sequence: Kueue admits a workload and creates its pods, and Karpenter sees the pods that do not fit and launches nodes for them.

The pairing changes what each side means:

  • Quota is the brake, and Karpenter has no queue of its own. Without Kueue, a burst of pending pods makes Karpenter try to launch nodes for all of them. With Kueue, only admitted workloads create pods, so the node count follows the quota. Karpenter’s own brake is spec.limits on a NodePool, which caps the resources the pool may consume, and the docs say that once a limit is exceeded provisioning is prevented until nodes terminate. The same page notes the check is eventually consistent, so a rapid scale-out can overrun it. A site sets the Kueue quota at or below the NodePool limit, so the queue holds work back before Karpenter has to refuse it.
  • A ResourceFlavor names a kind of node, and a NodePool makes it. A flavor selects nodes by label. A NodePool carries the labels and taints for one kind of node, such as a GPU type, and a pod whose requirements do not overlap the pool’s gets no node from it. Mapping one flavor to one NodePool keeps the two descriptions of the capacity in agreement.
  • Admission and capacity are separate facts. Kueue can admit a workload because quota is free while no node exists yet, and the pods wait until Karpenter has launched one. Kueue’s ProvisioningRequest admission check closes that gap by holding admission until capacity is confirmed, but the page describes it as an integration with the Kubernetes Cluster Autoscaler and names no other autoscaler. With Karpenter, admitted pods can therefore sit pending until a node is launched, and the docs I read give no check that holds admission for it.
  • Scale-down competes with admission. Karpenter consolidates nodes that are empty or underutilized after a consolidateAfter delay, and it evicts pods through the Eviction API with a taint on the node. A short gap between two queued jobs can turn a node off that the next job needs, so batch pools use a longer delay.

This section is reading, because the lab is three VMs with no cloud provider. The AWS autoscaling post covers an emulated version of node scaling.

Commands and results

Three drills, each run once. The cluster had the Ray head, MinIO, the model server and the monitoring stack running and idle.

Borrowing. team-a submitted six one-CPU jobs of 60 seconds. Kueue admitted four at once: its own 2 CPUs and 2 borrowed from team-b, which was idle. The other two waited in the queue.

Reclaiming. 23 seconds later team-b submitted two jobs. Within about 8 seconds Kueue had preempted two of the team-a workloads, with events saying “Preempted to accommodate a workload”, and team-b had two admitted workloads. Its pods did not start until about 42 seconds after it submitted, when the first two team-a jobs finished and freed real CPU. After that, the preempted team-a jobs were admitted again.

Excerpt of the borrowing and reclaiming drill Screenshot: part of the drill log. At the first line team-a has four workloads admitted but only two pods running. After team-b submits, the admitted counts move to two each, team-b’s pods sit pending, and they start running at 21:06:58 and 21:07:02. The times are from the machine that ran the drill.

All or nothing. I submitted a job with six one-CPU pods, which is more than the cohort’s 4 CPUs.

JobWhat happened
Six pods through KueueThe workload stayed pending, and no pods were created. Kueue’s message was that the request of 6 CPUs was more than the maximum capacity of 4.
Six pods with no queueTwo pods ran and four stayed Pending.
Four pods through KueueOne workload was admitted, and two pods ran while two stayed Pending for the first seconds.

Excerpt of the all-or-nothing drill Screenshot: part of the drill log. The six-pod job through Kueue has no pods at all and a pending workload. The same job with no queue has pods running and pending. The four-pod job through Kueue is admitted as one workload with two pods running and two pending.

Challenges

  • Admission is not capacity. In the borrowing drill Kueue admitted 4 CPUs of work, but only two pods could run. The nodes had about 1.4, 0.8 and 1.2 CPUs free, so only two of them could take a one-CPU pod, and the rest waited with “Insufficient cpu”. A quota that is bigger than the usable capacity, or capacity that is fragmented across nodes, makes the queue say yes when the scheduler cannot. In a fleet, quotas have to be set from what the nodes can really place.
  • Reclaiming quota did not reclaim CPU. The preempted jobs had admitted quota but their pods had not started, so stopping them freed nothing on the nodes. The team that was owed capacity waited for the jobs that were running to finish, about 34 seconds after its workloads were admitted.
  • All-or-nothing applies to the quota, not to placement. The four-pod job was admitted as a unit, and still started only half its pods. For a job that needs all its workers at once, such as distributed training, a half-started job holds resources and does no work. Kueue has a setting that waits for pods to become ready, which I did not try, and the schedulers listed below work at placement time.
  • The API changed between versions. My first manifests used the older v1beta1 version of the queue objects, and this release did not serve it. The newer one also renamed the cohort field, so the first apply failed with “unknown field”. Operators that ship custom resources need pinned versions and a manifest check on upgrade.

What this lab cannot show is GPU flavors, many teams, fair sharing between them over time, or jobs that need specific nodes.

Differences with a production setup

  • Resources. I used CPU and memory in one flavor. Production queues use a ResourceFlavor per kind of node, such as GPU models, and quotas in GPUs.
  • Gang scheduling. Starting all the pods of a job together is done at placement time by schedulers such as KAI Scheduler or Volcano. I did not try them.
  • Integration. Kueue works with other job types, including Ray, through the same queues. I used plain Jobs.
  • Fairness. Slurm has multifactor priority and fair-share, which weigh how much each team has used over time. My drill only showed quota and preemption.
  • Capacity. My three nodes are fixed, so quota was the only limit. In a cloud setup Karpenter adds nodes for admitted work, as described in the section above, and the quota has to be sized against the NodePool limits.

FAQ

What does Kueue do that the Kubernetes scheduler does not?

The scheduler places pods on nodes. Kueue decides which jobs may start at all, based on quotas and queues, and it holds a job’s pods back until it is admitted.

What is the difference between admission and scheduling?

Admission is the quota decision. Scheduling is placing a pod on a node that has room. A job can be admitted and still have pods that cannot be placed, which is what I saw.

What does borrowing mean, and when does a queue give it back?

A queue in a cohort can use another queue’s unused quota. When the owner submits work and is set to reclaim, Kueue preempts the borrowing workloads. The jobs go back to the queue and run again later.

How does this compare with Slurm?

Slurm queues jobs against nodes and tracks usage for fair-share. Kueue queues jobs against quotas, and the Kubernetes scheduler still decides where pods go. The two schedulers in this lab do not know about each other.

Why did the six-pod job never start?

Its request was larger than the whole cohort’s quota, so Kueue could never admit it. That is the intended result, and it is why no pods were created.

Next

Take the quiz to test your understanding: Part 5 quiz

The next post in Part 5 scales a workload on a metric from Prometheus with KEDA, and compares it with the CPU autoscaler from Part 4.

Comments