The foundations: Slurm, Kubernetes and the stacks that combine them

Goal

Parts 1 and 2 built a Slurm cluster and a Kubernetes cluster on the same three VMs. By the end I could tell what each one does, and I kept mixing up their words. Slurm has a drain and Kubernetes has a drain, and they are not the same thing. Kubernetes has a cordon and a taint, and Slurm has neither word. A GPU datacenter often runs both schedulers, so an operator or an onsite technician has to know which word applies to which system before touching a node.

This post is the map I wish I had at the start. It compares the two on the words that get confused, lists four ways they are combined, and reads what three public projects that run Slurm on Kubernetes say about node state.

There is no lab in this post. I did not install or run any of the three stacks, and I did not run a cordon-and-drain test across both schedulers on the three VMs. The lab figures come from Parts 1 and 2. Everything about Slinky, Soperator and SUNK is from their public documentation, and I say where a page was out of reach.

Terms used in this post

  • Workload manager: software that decides which work runs on which machines. Slurm and Kubernetes are both one.
  • Job: in Slurm, a script with a request for CPUs, GPUs, nodes and a time limit. It runs and ends.
  • Pod: in Kubernetes, the smallest thing it runs. A service is kept alive as pods.
  • Node: one machine in either system. Each system keeps its own record of the node’s state.
  • Drain (Slurm): stop placing new jobs on a node and let the running ones finish. The node reaches drained when the last job ends.
  • Drain (Kubernetes): cordon the node, then evict the pods on it, within any disruption budget.
  • Cordon: mark a Kubernetes node so no new pods are placed on it. Running pods stay.
  • Taint and toleration: a taint on a node repels pods, and a toleration on a pod lets it through. The effects NoSchedule and NoExecute are in Part 2.
  • Partition: in Slurm, a named group of nodes with its own limits. It is the nearest thing to a queue.
  • Reservation: in Slurm, nodes set aside for a window of time, for example for maintenance.
  • Gang scheduling: starting all parts of a job at the same moment, or not at all. A multi-node training job needs it.
  • Operator: a Kubernetes controller that manages another piece of software from one custom resource.
  • Pod condition: a status field on a pod that other tools can read.

What I built

A comparison and a decision table. No machines.

Four ways to run Slurm and Kubernetes together Diagram: separate machines, the same machines, Slurm inside Kubernetes, and Slurm placing Kubernetes pods.

How it works

Two different questions

Slurm answers when and where a finite job runs, and for how long. A job asks for resources and a time limit, waits in a queue, runs and ends. Kubernetes keeps a stated number of copies of a service running and replaces them when something fails. A pod is not expected to end.

That difference explains most of the vocabulary. A scheduler for jobs can wait for a running job to finish before it takes a node away. A scheduler for services usually moves the service. Part 1 showed the first and Part 2 showed the second.

The same words, different meanings

IdeaSlurmKubernetes
Stop new work on a nodedraincordon
Also move the work offNot part of drain. Jobs finish, or an administrator ends themdrain: evicts pods, within any disruption budget
Node is unusabledown, and down* when the controller cannot reach itNotReady, then NoExecute taints that evict pods after their toleration time
Keep some work off a node on purposePartitions, node features, reservationsTaints and tolerations, with labels and selectors
Set nodes aside for a maintenance windowA reservationNo equivalent. An operator cordons or taints, and nothing starts in advance to avoid it
All parts of a job start togetherNative: a job gets all its nodes or waitsNot in the default scheduler. A queue such as Kueue can admit a job as a whole, and in that lab the guarantee applied to quota and not to placement
Who may use how muchFair-share, QoS, accountingQuotas and queues, added by tools such as Kueue
Record of past worksacct, from slurmdbdNo long-term record built in
Check node healthA health-check program on each nodeProbes, node conditions, and whatever tool reads them

Two rows cause most of the mix-ups. A Slurm drain is closer to a Kubernetes cordon than to a Kubernetes drain: both stop new work and leave running work alone, and only the Kubernetes drain goes on to evict. And a cordon is a taint. The control plane sets a NoSchedule taint named node.kubernetes.io/unschedulable for you.

What the labs measured:

  • Slurm drain: a 40 second job was running when I drained the node. The node became drained 36 seconds later, when the job ended, and a job pinned to it stayed PENDING with ReqNodeNotAvail until I resumed it.
  • Slurm down: after I stopped slurmd, the node showed idle* at 45 seconds and down* at about 228 seconds. SlurmdTimeout was 300.
  • Kubernetes silent node: NotReady at 44 seconds, and a replacement pod ready at 345 seconds with the defaults, or 76 seconds with a 30 second toleration.
  • Kubernetes drain: with a disruption budget of minAvailable: 3 on three replicas, the drain evicted nothing and timed out after 40 seconds, leaving the node cordoned. With minAvailable: 2 it finished in about a second.

Four ways to combine them

A. Separate machines. One cluster of each kind. This is the simplest and wastes the most hardware when one side is idle.

B. The same machines, two schedulers. This is the lab. slurmd and the kubelet run side by side on each VM. Each scheduler thinks it owns the node’s memory: Slurm was told 2800 MB, and Kubernetes saw the full 2984 MB as allocatable. Slurm still killed a job that went over its cgroup limit while the kubelet was running, so isolation held and accounting did not. By design, neither scheduler reads the other’s node state, so a Slurm drain does not stop Kubernetes from placing pods on the node and a Kubernetes cordon does not stop Slurm from starting jobs. I did not test that on the lab, and I say so in the list of things not done below.

C. Slurm inside Kubernetes. An operator turns one custom resource into a working Slurm cluster. The Slurm daemons run as pods, and Kubernetes handles their lifecycle. This is what Slinky’s slurm-operator, Soperator and SUNK each do in their own way.

D. Slurm places Kubernetes pods. A bridge or a scheduler plugin lets Slurm’s scheduler decide where some Kubernetes pods go, so that jobs and pods share one view of capacity. Slinky’s slurm-bridge and SUNK’s Pod Scheduler are examples.

The three stacks, from their documentation

SlinkySoperatorSUNK
FromSchedMD, the maintainers of SlurmNebiusCoreWeave
What it isA set of projects. slurm-operator runs Slurm components as pods, and slurm-bridge lets Slurm act as a scheduler for Kubernetes workloadsA Kubernetes operator that turns a SlurmCluster resource into a Slurm clusterA Helm-deployed Slurm on Kubernetes, with a scheduler plugin and a Kubernetes Pod Scheduler
Slurm nodeA pod in a NodeSet, running slurmdA worker NodeSet podA pod running slurmd
Node stateSlurm node states are applied to each NodeSet pod as pod conditionsPassive and active health checks on GPU, network, storage and system signalsThe docs say Slurm events are tracked in Kubernetes and Kubernetes events are propagated to Slurm. I did not find the details
MaintenanceSlurm nodes are marked drain before termination for scale-in or upgradeFailed nodes are drained and replaced automatically. Upgrades and resizing come from changes to the resourceNon-SUNK pods are invisible to Slurm, and the docs warn that running them on Slurm nodes can cause resource conflicts and unexpected node drains
NotableScale to zero for workers and login nodes. Components can also live outside KubernetesA shared root filesystem for login and worker nodes, called the jail. InfiniBand-aware placementA pod can ask for the SUNK scheduler by name, and then Slurm places it with a placeholder job

Sources: the slurm-operator documentation for Slinky, the Soperator repository, and CoreWeave’s SUNK page and its page on scheduling Kubernetes pods. The Slinky page I read is the v0.4.1 documentation, and Slinky has since reached a 1.x release, so details may have changed.

What maintenance means in each arrangement

ArrangementWhat a technician’s maintenance window needs
A. Separate machinesWhichever of drain or cordon and drain belongs to the system on that node
B. Same machinesBoth: a Slurm drain so that no jobs start, and a Kubernetes cordon or drain so that no pods land
C. Slurm in podsA Kubernetes node action, and Slurm hears about it only if the operator passes it on. Slinky says it drains Slurm nodes before it terminates them for scale-in or upgrade. I could not find what it does when the Kubernetes node itself is drained
D. Slurm places podsWhichever system placed the work. A pod that skipped the Slurm scheduler is invisible to Slurm

Commands and results

There were no commands for this post. What I did:

  • Compared the vocabulary of Part 1 and Part 2 against the lab logs, and used the numbers above.
  • Read the public documentation for the three stacks and wrote down what each says about node state and maintenance.
  • Wrote down what I could not answer from the documentation, listed under Challenges.

Challenges

  • Not run: the cross-scheduler drill. The test I wanted was a cordon in Kubernetes and a drain in Slurm on the same VM, then checking what the other scheduler does. I could not reach the VMs from the machine I was working on, so I did not run it. The statement that neither scheduler sees the other’s state comes from the design, and from the memory accounting I did measure.
  • A page I could not read. CoreWeave’s SUNK overview page asks for an access code. What I report about SUNK is from its other public pages and a search summary, and it is thin on node state.
  • Versioned documentation. The Slinky pages exist for several releases, and the one I read is not the newest. Anything that depends on how it handles a node drain should be checked against the version in use.
  • Vendor documentation describes the intent. A page that says failed nodes are drained and replaced automatically does not say how long that takes or what happens to the job on the node.
  • The words overlap. Slinky’s slurm-operator and Soperator both have a NodeSet. A Slurm node, a Kubernetes node and a pod can all be called a node in the same paragraph.

Differences with a production setup

  • The lab has three small VMs and no GPUs. Production Slurm on Kubernetes has thousands of GPUs, an InfiniBand fabric, shared storage for the training data and checkpoints, and a team that owns each of those.
  • The stacks above lean on other pieces: a GPU operator for drivers, a network plugin, a storage class, and an observability stack. I did not look at those.
  • Production teams usually pick one arrangement per cluster. Arrangement B is how this lab got built, and it is the one most likely to cause an incident, because the two schedulers do not share what they promise.

Use cases

The lab was three VMs, so none of this was run on GPU hardware. The scenarios come from how the mechanisms are documented to work.

Choosing the scheduler for a customer (solutions architect). A customer who runs long multi-node training jobs, wants fair-share between teams and needs accounting is describing Slurm. A customer who serves a model and wants it kept alive is describing Kubernetes. A customer with both is a case for arrangement C or D, or for two clusters. The question that decides it is who takes nodes out for maintenance and how.

A shared node needs a part swapped (onsite technician and operator). In arrangement B the operator does both: drain the node in Slurm so nothing starts, and cordon and drain it in Kubernetes. The technician is told when both are done. Doing one leaves the other scheduler free to place work on a node that is about to be opened. In arrangement C the operator works at the Kubernetes layer and checks that the Slurm node reached drain first.

A GPU fault on a training node (on-call). The first question is which system owns the node. If it is Slurm, scontrol update with a reason sets the drain, and the reason is what the next shift reads in sinfo -R. If it is Kubernetes, the operator cordons, or applies a NoExecute taint if the pods must leave now, as in Part 2. In arrangements C and D, the stack’s own health checks may already have done it, so the operator checks before repeating it.

A node goes silent (on-call). Slurm waits for SlurmdTimeout and then marks the node down*. Kubernetes marks it NotReady and evicts after the pods’ toleration time. The two timers are set separately, so on a shared node the same failure shows up on two different clocks. In my lab the times were 228 seconds against 44 and 345 seconds.

Upgrading the platform (platform engineer). In arrangement C an upgrade is a change to a resource, and the operator drains Slurm nodes before it replaces them. The cost is that the Slurm cluster’s lifecycle is now tied to the Kubernetes cluster’s. A disruption budget that blocks eviction, as in Part 2, can stall a Slurm node’s replacement.

Adding a service next to training (solutions architect). A monitoring agent or an inference service has to land on nodes Slurm also uses. In arrangement D, the pod names the Slurm scheduler and Slurm counts it. In arrangement B, nobody counts it, which is the risk the lab measured.

FAQ

What is the difference between a Slurm drain and a Kubernetes drain?

A Slurm drain stops new jobs and lets the running ones finish. A Kubernetes drain cordons the node and then evicts its pods. The closest Kubernetes action to a Slurm drain is a cordon.

Is a cordon the same as a taint?

A cordon is a NoSchedule taint that the control plane sets for you. A taint you set yourself can use any key and any of three effects, and is for standing rules.

What is Slurm on Kubernetes?

Slurm’s daemons, slurmctld, slurmd and the rest, run as pods, and an operator manages them from one resource. Users still submit jobs with sbatch and srun.

What is the difference between Slinky, Soperator and SUNK?

Slinky is from SchedMD and includes an operator and a bridge that lets Slurm schedule Kubernetes pods. Soperator is an open source operator from Nebius, which Nebius also offers as a managed service. SUNK is CoreWeave’s own implementation, with a Kubernetes Pod Scheduler that goes through Slurm. These are summaries of their documentation, not a comparison I tested.

Why not just use Kubernetes for training jobs?

You can, and teams do. What Slurm brings is gang scheduling, fair-share, accounting and a long-standing model for multi-node jobs. Queue tools such as Kueue add some of that to Kubernetes, and in that lab admission was a quota decision that did not guarantee placement.

Why not run both on the same machines?

The lab did, and it works for a small cluster. Each scheduler counts the whole node, so they can overcommit it, and each one’s maintenance state is invisible to the other.

Next

Take the quiz to test your understanding: Slurm and Kubernetes quiz

Part 1 builds the Slurm cluster, and Part 2 puts Kubernetes on the same three VMs.

Comments