3 Industry usedone · reading only
6 Bare metaldone
Emulated BMCs·Rack power plan·Drift and burn-in·NetBox inventory·Hardening and Secure Boot·Network boot with PXE·RAID and management network
7 Cloud + GPUemulated done · real run pending
AWS autoscaling·GPU fleet monitoring·GPU scheduling·NUMA topology
Goal
Slurm is the scheduler, or workload manager, behind many HPC and GPU clusters. I started here because it shows scheduling and accounting in a form small enough to hold in my head, before Kubernetes adds its own vocabulary.
What I built
Three Debian 12 VMs on Proxmox: one controller (slurm-ctl) and two compute nodes (lab-node1, lab-node2), 2 vCPUs and 3 GB each, running Slurm 22.05.8. Everything is configured by one idempotent Ansible playbook, so a second run changes nothing.
Slurm is a queue plus a placement engine. You say what a job needs, which is its resource request (CPUs, memory, nodes, time limit), and the controller decides when and where it runs. The other pieces make that work across several machines:
- munge lets machines authenticate each other.
- slurmd runs the job.
- cgroups enforce the limits.
- NFS makes files visible wherever the job lands.
- slurmdbd keeps a permanent record.
Diagram: the three VMs and what runs on each.
How it works: one job from start to finish
Diagram: the second compute node is not drawn.
sbatch hello.sbatchtalks to slurmctld onslurm-ctl, carrying a munge credential: a signed, timestamped token.- slurmctld records the job as
PENDING. Its scheduler looks for nodes with free cores and memory. - It tells slurmd on the chosen node to start the job, again authenticated with munge.
- slurmd starts a slurmstepd, puts the job in its own cgroup, and runs the script as your user. That is why user IDs must match on every node.
- The script writes its output to
/shared, which has the same path on every node because of NFS. - When the job ends, the record goes to slurmdbd and then MariaDB.
sacctreads it back from there.
What each piece does, and what breaks without it
| Piece | Why | If missing or wrong |
|---|---|---|
| munge, shared key | Nodes trust each other without passwords | Auth errors; nodes show down |
| Same uid/gid on every node | A job runs as the same user everywhere | Wrong file ownership; munge can fail |
| chrony (time sync) | munge tokens expire quickly | ”Credential expired” errors |
Identical slurm.conf | One shared cluster definition | Nodes refuse to register |
Controller-local StateSaveLocation | The queue survives a restart | Queue lost on restart |
NFS /shared | Job files at the same path anywhere | Jobs fail on the wrong node |
| cgroups | Enforce the CPU and memory requested | A greedy job starves its neighbours |
| slurmdbd + MariaDB | Accounting | The cluster runs, but sacct is empty |
The NFS share is for job data, not for Slurm’s own state. The controller keeps that state on its own local disk, in the directory set by StateSaveLocation (/var/spool/slurmctld).
Diagram: three separate stores, and what happens if each one is lost.
What a node checks before it trusts a request
Diagram: the credential, the three checks on the receiving node, and the failure each one produces.
Names and addresses
Diagram: the default cloud-init behaviour and what the lab does instead.
Running jobs in containers: Pyxis and Enroot
Large GPU clusters rarely run a job directly on the host’s libraries. Users bring their own software stack as a container image, and Slurm has to start it. The usual pairing is Enroot, a tool that turns a container image into an unpacked root filesystem and runs a process in it without a container daemon, and Pyxis, a Slurm plugin that adds container options to srun and sbatch. Pyxis is a SPANK plugin: slurmd loads it from plugstack.conf and calls it as each job step starts, and it asks Enroot to import the image, create the container and run the step inside it.
--container-imagetakes a registry name such aspython:3.12-slimor the path to a squashfs file made earlier withenroot import.--container-mountsbind-mounts host paths, such as the shared filesystem, into the container.--container-namekeeps the unpacked container on the node so a later step can reuse it.
Nothing about scheduling changes. Slurm still allocates the nodes, the cgroups still enforce the limits, and sacct still records the step. The container is how the process sees its filesystem.
Commands and results
-
A two-node
sbatch -N2job completed (exit code 0:0) on both compute nodes, wrote its output to/shared, andsacctreturned the job and its steps. The cgroup paths in the job’s output named the real nodes, which is how I confirmed where it ran. -
Drain drill: I drained a node while a job was running on it:
- 04:35:36, a 40-second job is running on
lab-node1. - 04:35:40, I set the node to drain. State becomes
draining. - A second job with no node preference ran on
lab-node2. A third, pinned tolab-node1, stayedPENDINGwith reasonReqNodeNotAvail. - 04:36:16, the running job finished and the node became
drained, with my reason kept. - 04:36:19, I resumed the node. At 04:36:21 the pinned job ran.
So drain lets running work finish and holds new work until the node returns.
- 04:35:36, a 40-second job is running on
-
Unresponsive node drill: I stopped
slurmdonlab-node1and checkedsinfoevery ten seconds.SlurmdTimeoutis 300 seconds on this cluster.- For the first 34 seconds the node still showed
idle. By 45 seconds it showedidle*: the asterisk means the controller is not getting answers from the node, and the node still counts as usable until the timeout. - About 228 seconds after I stopped it, the state became
down*with the reasonNot responding. That is under the 300 seconds. The slurm.conf documentation says the controller waitsSlurmdTimeoutseconds for slurmd to respond before it sets the node to DOWN, and that it pings the daemons over a hierarchical communication tree. I read that as a wait counted from the last time the node answered, which was before I stopped the daemon, so the visible time is shorter than the setting. - When I started
slurmdagain the node wasidleat the next check, one second later, with noscontrolcommand, becauseReturnToService=2lets a node that registers with a valid configuration return by itself.
So
down*is a node the controller cannot reach, and the reason field and theslurmdlog on that node are the first places to look. - For the first 34 seconds the node still showed
-
Containers with Pyxis and Enroot, installed on both compute nodes. The same two-node step,
srun -N2 --container-image=python:3.12-slim hostname, printed both node names, and/etc/os-releaseinside the step read Debian 13 while the hosts are Debian 12, which shows the step ran in the image. Timings from one run each on 2 vCPU VMs:- no container: 2.3 s
- registry image: 20.3 s on the first run and 27.2 s on the second. Pyxis printed
imported docker imageon every run, so the registry name is imported again each time, and the layer cache saves only the download. - squashfs file on
/shared: 3.0 s. The 82 MB file took 7 s to build withenroot import. - named container: 4.6 s to create and 0.7 s to start again on the same node.
- A bind mount of
/shared/enroot-demoas/datalet the container write a file that appeared on the host with the job user’s ownership. A two-nodesbatchscript with the squashfs image completed in 1 s insacct.
So a container adds seconds if the image is a file on shared storage and tens of seconds if every step pulls from a registry, which is why GPU sites import once and run from a file.
-
Ansible idempotency: two consecutive full runs reported
changed=0on all three hosts.
Diagram: the drain drill, using the timestamps from the log.
Challenges
Things that went wrong along the way:
- A wrong diagnosis of mine. A node refused SSH and I blamed a host-key conflict. It was simply a new host that had not been accepted yet under non-interactive SSH.
- NFS mount denied. Ansible handlers run at the end of a play, so the export was not reloaded before the client tried to mount.
meta: flush_handlersfixed the ordering. - A playbook that was not idempotent. Two roles set the same directory to different modes (0750 and 0755), so every run flipped it. I aligned them.
/etc/hostsand cloud-init. cloud-init rewrites/etc/hostson every boot and maps the node’s own name to127.0.1.1, which confuses Slurm. I turned that off and let Ansible own the file. That fix did not hold through a power cycle. The hypervisor generates the cloud-init data again at power-on and its setting overrode mine, so after a hard power-off a node could not resolve the controller’s name and Slurm kept itDOWN. I found this later, in Part 4, and fixed it by disabling cloud-init on the nodes in the Ansible role. It then held through three power cycles. A fix is not finished until it has been tested across a power loss.- Container runtime state needs a directory that exists. Enroot keeps its per-user runtime files under
$XDG_RUNTIME_DIR, which only exists for users with a login session. A batch job on a node where nobody is logged in failed withmkdir: cannot create directory '/run/user/1000', and with Pyxis markedrequiredthe whole step failed. SettingENROOT_RUNTIME_PATHin/etc/enroot/enroot.confto a path under/tmpfixed it. Without that setting a node that works for one person can fail for the next job. - A named container lives on one node.
--container-nameon a second node does not find the container made on the first, so steps that must share a container have to run where it was created.
Differences with a true HPC cluster like Meta’s
This lab is three small VMs, so it says little about how a large cluster behaves. Meta’s published numbers show the gap. In March 2024 Meta described two clusters of 24,576 NVIDIA H100 GPUs each, and its earlier AI Research SuperCluster was planned to connect 16,000 GPUs. The lab has none.
- Network fabric. My nodes share one virtual network on a home LAN. Meta built one of its clusters on RoCE with Arista 7800 switches and the other on NVIDIA Quantum2 InfiniBand, and says it changed its internal job scheduler to place jobs with network topology awareness. Slurm can do this with a
topology.conffile. I haven’t configured it, because my nodes have no switch layout to describe. - Storage. My
/sharedis one NFS export on the controller. Meta describes a flash-optimised distributed store behind a FUSE interface for checkpoints, plus a parallel NFS deployment built with Hammerspace, at exabyte scale. - Scheduling policy. I did not try priorities and fair-share or preemption.
- Controller failure. The lab has one controller. Slurm supports standby controllers through several
SlurmctldHostentries, where the first is primary and the rest wait as backups. I haven’t tried that, or what happens when the controller itself fails. I also haven’t tested whether the/etc/hostsfix survives a reboot. - Accounting and limits. The accounting database is one MariaDB on the controller with no separate backup. I haven’t set quotas, and I have no monitoring beyond reading
sinfoand logs. - Experiments not yet run. Stopping munge, stopping slurmd and unmounting
/sharedare written down as predictions, and I haven’t run them. - Containers at scale. I ran Pyxis and Enroot on two nodes with an 82 MB image and no GPU. A GPU cluster adds the NVIDIA container hooks that expose the GPUs inside the container, images of many gigabytes, and a parallel filesystem to hold the squashfs files, none of which I have here.
- Slurm and Kubernetes together. Some sites queue batch work on Kubernetes with Kueue, which admits jobs against quotas, and the Slinky project runs Slurm itself on Kubernetes. I ran neither, and the two schedulers in this lab do not know about each other.
Meta’s post describes an internal scheduler, not Slurm, so this comparison is about scale and design and not about one tool.
FAQ
What is chrony?
chrony is an implementation of NTP, the Network Time Protocol, which keeps a machine’s clock in step with time servers (chrony). The cluster needs it because munge credentials carry a timestamp and expire after a time-to-live, 5 minutes by default. The fastest and slowest clocks in the cluster have to be within that window. Slurm’s quick start says the same: munge requires synchronized clocks, usually through NTP. In the lab, Ansible installs and starts chrony on all three VMs.
What is cloud-init?
cloud-init is the standard tool for first-boot setup of a cloud image. On boot it reads data supplied by the platform, such as the hostname, network settings and SSH keys, and applies it. My VMs use the Debian 12 cloud image, and the creation script passes each VM’s static address and first SSH key through cloud-init. Its side effect is the one in the Challenges list: it rewrites /etc/hosts on every boot and maps the node’s own name to 127.0.1.1, which confused Slurm. I first turned that off with manage_etc_hosts: false, which did not survive a power cycle (see Challenges), and now disable cloud-init on the nodes and let Ansible own the file.
What is saved in slurm.conf, and must the hardware be the same?
slurm.conf is the cluster definition, and it is identical on every node. In this lab it holds the cluster name and controller host, the authentication type (munge), the process-tracking and task plugins (cgroup), the state directory, the scheduler (backfill) with cores and memory as consumable resources, the accounting settings (slurmdbd), one line per compute node, and the partition.
The hardware does not have to match. The file is the same everywhere, but each NodeName line describes that node’s own CPUs and memory, so a cluster can mix node types, grouped into partitions or labelled with features and GPUs. My two nodes are identical because my template gives every compute node the same values. What a node must do is register with at least the resources the file says it has. If it registers with fewer, Slurm marks it INVAL and it takes no work until slurmd is restarted with a valid registration.
Which node states exist besides draining?
These come from the sinfo documentation, which covers a newer Slurm than my 22.05.8, so a few of them may not exist in my version. In my drills I only saw idle, draining, drained, idle* and down*.
| Group | States | Meaning |
|---|---|---|
| Working normally | IDLE, ALLOCATED, MIXED, COMPLETING | free, fully used, partly used, or finishing the jobs it had |
| Unavailable | DOWN, DRAINED, DRAINING, FAIL, FAILING, INVAL, UNKNOWN, FUTURE | failed or unreachable, removed by an administrator, expected to fail, registered wrongly, not yet checked by a restarted controller, or not yet configured |
| Reserved or planned | RESERVED, MAINT, PLANNED, BLOCKED | in a reservation, in a maintenance reservation, set aside by the backfill scheduler for a higher-priority job, or blocked by an exclusive topology job |
| Power | POWER_DOWN, POWERING_DOWN, POWERED_DOWN, POWERING_UP | powering off, off, or powering on |
| Reboot | REBOOT_REQUESTED, REBOOT_ISSUED | a reboot asked for, or sent to the agent |
A suffix can follow a state: * means the node is not responding (the unresponsive node drill above shows it appearing on idle first and on down after SlurmdTimeout), ~ means it is powered off, # means it is powering up or being configured, and $ means it is in a maintenance reservation.
Next
Take the quiz to test your understanding: Slurm quiz
Part 2: the same machines, running Kubernetes.
Comments