Slurm and Kubernetes in industry: who runs what, and why

Goal

Parts 1 and 2 built a Slurm cluster and a Kubernetes cluster on the same VMs, and the Part 0 comparison sorted out their vocabulary. This post asks the next question: at the scale where GPU clusters are bought, which one do people actually run, and why? A large cluster that runs Slurm for training, Kubernetes for a few utility nodes and a vendor’s cluster manager on top is an ordinary arrangement, and the reasons are partly technical and partly cultural.

Most of the later parts of this series are about Kubernetes: the ML platform, the platform layer and the Advanced Kubernetes track. The scorecard below is the map of where Kubernetes needs help to match what Slurm does natively, and which post covers each gap.

Terms used in this post

  • Gang scheduling: starting all the pods or tasks of a job together or not at all, which multi-node training needs.
  • Fair-share and QOS: Slurm policies that weigh how much each user or account has used and cap what a class of jobs may take.
  • Preemption: stopping a lower-priority job so a higher-priority one can run.
  • Topology-aware scheduling: placing a job on nodes that are close in the network or NVLink fabric.
  • SPANK plugin: a Slurm extension point. Pyxis, the container plugin in the Slurm post, is one.
  • Slurm on Kubernetes: running Slurm’s daemons and compute nodes as Kubernetes pods. Named implementations are Slinky (SchedMD and NVIDIA), Soperator (Nebius), SUNK (a managed service) and Slonk (Character.ai).
  • IMEX: NVIDIA’s service that maps GPU memory across nodes in an NVLink domain, described in the NVLink post.

What I built

Nothing new. The two lab clusters and the posts about them are the working examples, and this post is reading. Most of the sources are written by vendors who sell one of the two approaches, so I note when a claim comes from the seller.

How it works

Scorecard

The legend: 🟢 easy support, 🟡 okay support (needs a plugin, operator or extra configuration), 🔴 hard support. The rows are adapted from public Slurm-versus-Kubernetes comparison tables, including those from Mevasis and Clusterbid, plus the comparisons from Nebius, SkyPilot and Hyperstack.

CriterionKubernetesSlurmWhat the industry evidence saysWhere the lab covers it
Queueing, gang scheduling, topology and preemption🟡🟢Slurm has these natively. Kubernetes needs Kueue, Volcano or JobSet added. This is the clearest agreement across public comparisons.Kueue, GPU scheduling
MPI and multi-node training🟡🟢Slurm launches MPI and multi-node jobs directly. Kubernetes goes through operators (MPI, PyTorch, Kubeflow), which add a layer to configure and debug.Ray
Low-latency network (InfiniBand/RDMA)🟡🟢Slurm uses InfiniBand directly. Kubernetes can too, but needs device-plugin and network configuration. One public table calls Kubernetes support “limited” while others say it works with care, so it is rated okay.InfiniBand and RoCE
Multi-tenant management🟢🟢Fair-share and QOS policies versus RBAC, namespaces and quotas. Both are mature. This row is about sharing policy, not security isolation.RBAC and tenancy
Container support🟢🟡Kubernetes is built around containers. Slurm usually reaches them through Pyxis/Enroot or Apptainer, which works but is an add-on.Slurm post, Pyxis and Enroot
Shared filesystem🟡🟢Slurm clusters typically mount a parallel filesystem (Lustre, GPFS) on every node. Kubernetes uses persistent volumes, where parallel filesystems are more awkward. Shared read-write volumes narrow the gap.Data movement
Auto-scaling🟢🔴Kubernetes scales nodes and replicas natively. Slurm clusters are mostly static and burst to the cloud through plugins.Kubernetes post, Kueue and Karpenter, KEDA
Inference and serving🟢🔴Kubernetes is designed for always-on services. Slurm is awkward for APIs and continuous serving, though some HPC sites do it.Model server
Setup complexity🔴🟢Slurm is lighter to stand up. Kubernetes adds layers (CRDs, device plugins, drivers). Depends on whether a platform team already runs clusters.Both lab posts

Who uses what for large training

Which scheduler large sites use for GPU training Diagram: the camps, from the public sources below.

Evidence quality varies: some rows are the organization’s own paper, some are a third-party write-up, and the vendor case studies are written by the vendor.

WhoWhat they useNotes
Meta (Llama 3)In-house scheduler, MASTThe paper says training jobs are scheduled using MAST.
Ai2 (OLMo 2)In-house workload manager, BeakerThe paper has a section on Beaker.
OpenAIKubernetes, 7,500 nodes, MPI in pods, Coscheduling pluginDated January 2021.
AnthropicKubernetes on GKE with TPUs, with a custom scheduler that decides which workloads to evict and how to reconfigure topologyThird-party write-ups of talks, one of them. TPUs, not GPUs.
Hugging Face (SmolLM3)384 H100 GPUs on 48 nodes for 24 daysThe model blog and the training playbook describe the run.
Character.aiSlonk: Slurm on KubernetesResearchers wanted a reliable scheduler with fair queues and gang scheduling, and the infrastructure needed Kubernetes for orchestration, health checks and autoscaling. From Character.ai’s own post, read through a secondary summary.
MistralSlurm on Kubernetes through the managed SUNK service, training and inference in one clusterDescribed in the service vendor’s case study, not in a statement from Mistral.
PhotoroomSlurm on an H200 InfiniBand cluster for multi-node training, Kubernetes for production inference, one client layer (SkyPilot)Its post says inference wins when it needs a GPU back: the batch job is evicted and relaunched later. Each team owns its scheduler.

There are three camps (in-house, Kubernetes, Slurm) and a hybrid camp. I found no major lab stating “we chose Slurm over Kubernetes” in its own words.

The practitioner split

  • A Red Hat engineer in HPCwire (15 May 2026) recommends Slurm for pre-training and Kubernetes for inference, concedes that some teams pre-train on Kubernetes, and frames the divide as deep-seated tribal faultlines in the technical community, operations on one side and the machine-learning researchers on the other. This is one practitioner’s opinion.
  • SkyPilot says each is weak where the other is strong, so teams train on Slurm and serve on Kubernetes, which means two stacks and GPUs stranded in one pool. It is a vendor of a layer over both.
  • SkyPilot’s migration post names a cost: a 6-line Slurm script becomes 60 or more lines of Kubernetes YAML, salloc becomes pod exec and port-forwarding, and a shared NFS home is not automatic.
  • The dissent is OpenAI’s 2021 post: Kubernetes alone at very large scale, after adding gang scheduling. It is dated.

Bridges: running both

  • Slurm as pods: the managed SUNK service, Nebius Soperator (its rationale says Slurm is not cloud-native and that Kubernetes lacks a fit for training workloads), Character.ai’s Slonk, and SchedMD and NVIDIA’s Slinky.
  • NVIDIA’s own report: Slinky in production on clusters of more than 8,000 GPUs for LLM training and multi-node inference, with GPU communication performance matching non-containerized Slurm, while keeping Slurm scripts and fair-share and gaining Kubernetes health remediation. Vendor-authored.
  • The limit of a bridge: in NVIDIA Mission Control, moving a node between Run:ai and Slurm is a documented administrator procedure, so the split between the two is set by people and not rebalanced by demand.
  • Kubernetes-side queuing: Kueue does admission and quota and does not replace the kube-scheduler, which is what the Kueue post showed.
  • Client layers: SkyPilot and dstack give one interface over Slurm, Kubernetes and SSH machines.

Inference

  • Always-on serving is Kubernetes-native. KServe with llm-d (prefix-cache-aware routing, prefill and decode split into separate pods) is the pattern the CNCF material on llm-d describes, and the same CNCF whitepaper describes Slinky as the tool that brings Slurm jobs, gang scheduling and MPI into Kubernetes and lists Volcano and YuniKorn as other batch schedulers.
  • Sharing GPUs: a GKE tutorial puts a high-priority inference Deployment and a low-priority fine-tuning Job on one cluster under Kueue. Preemption pauses training when serving needs the GPUs, and freed accelerators go back to training as demand drops. Vendor-authored.
  • Slurm-hosted serving exists, mostly at academic HPC centers. The RAMSES system at the University of Cologne pairs a Kubernetes web API with vLLM launched as Slurm jobs, with about 500 ms of added latency in an initial benchmark.

GB200 NVL72 and SuperPOD scheduling

A rack-scale NVLink domain adds a placement constraint that both schedulers have to learn: a job’s pods or tasks must sit inside one IMEX domain. Everything in this block is from NVIDIA documents and blogs.

  • DGX SuperPOD GB200 uses Mission Control, which is Base Command Manager plus Run:ai, so Slurm and Kubernetes access come in one stack.
  • Slurm: NVIDIA’s block scheduling post recommends one block per 18-node NVL72 domain and the --segment option for sub-rack placement, and says the switch/nvidia_imex plugin manages IMEX channels so jobs sharing an NVLink domain are isolated from each other.
  • Kubernetes: the NVIDIA DRA driver’s ComputeDomain creates and tears down an IMEX domain for each workload, and pods must share an NVLink partition, which the nvidia.com/gpu.clique node label identifies.
  • Topograph is NVIDIA’s tool that supplies network and NVLink-domain topology to Slurm and to Kubernetes.

Commands and results

There is nothing to run. The lab can add one concrete cross-check to the scorecard: on the Slurm cluster a container job needed one plugin, and the cold, warm and file-based timings are in the Slurm post. On Kubernetes the same job needs no plugin, and an equivalent queue needs Kueue, as in the Kueue post.

Challenges

  • Two state systems, one node. A node can be drained in Slurm and cordoned in Kubernetes, and the two do not know about each other. The Part 0 post lists the words and what each does.
  • A static split strands GPUs. If a site gives some nodes to Slurm and some to Kubernetes, demand on one side cannot use idle nodes on the other until an administrator moves them.
  • Preemption needs a checkpoint. Both the Photoroom and GKE examples evict batch work when serving needs the GPUs. That is only safe if the training job saves state often enough to restart.
  • Topology labels go stale. Slurm blocks and Kubernetes clique labels describe the NVLink domain. A node swap or an NVSwitch tray replacement changes what they should say, and a wrong label gives a job a placement that does not work.
  • Vendors sell a side. Most of the comparisons are written by a company selling Slurm on Kubernetes, a Kubernetes layer or a neutral client. The sources agree on the clearest rows, queueing and serving, and differ on the rest.

Differences with a production setup

  • Scale. The lab has three nodes per scheduler. The sources describe clusters from 48 nodes up to many thousands of GPUs.
  • Evidence. These are public posts and papers, many by vendors. None of the scale or overhead figures were measured here.
  • Slinky, Soperator and SUNK were not run in the lab. The Part 0 post reads their documentation for node states.
  • The scorecard is a summary of other people’s comparisons, so it says where the consensus is and not what a particular site should pick.

FAQ

Which one do the big labs use?

Public evidence shows three camps: in-house schedulers (Meta’s MAST, Ai2’s Beaker), Kubernetes (OpenAI in 2021, Anthropic on GKE with TPUs) and Slurm, plus a hybrid camp that runs Slurm on Kubernetes (Character.ai, Mistral on SUNK) or keeps the two side by side (Photoroom).

Why do training teams like Slurm?

It queues, gang-schedules, preempts and places jobs by topology natively, launches multi-node jobs directly and uses InfiniBand without extra configuration. A training team can run a script and wait.

Why does inference go to Kubernetes?

Serving is always on, scales with traffic and needs health checks and rollouts, which Kubernetes was built for and Slurm was not.

Can one cluster do both?

Yes, with Slurm on Kubernetes or with Kueue on Kubernetes. Each moves some work to the operator or the queue, and a static split of nodes between the two is common.

Does anything change with NVL72 racks?

Placement has to respect the IMEX domain. Slurm does it with blocks and an IMEX plugin, and Kubernetes does it with a ComputeDomain and a clique label.

What does this reading not show?

Any measurement of my own on scale or overhead. It shows where the public evidence agrees and where it is one vendor’s view.

Repeating with real hardware

Coming soon. The nearest repeat is running Slinky on the lab cluster, with Slurm nodes as pods next to the existing Kubernetes workloads.

Next

Take the quiz to test your understanding: Part 3 quiz

Part 4 starts the ML platform on Kubernetes with a model server.

Comments