A three-node Slurm cluster from scratch

Goal

Slurm is the scheduler, or workload manager, behind many HPC and GPU clusters. I started here because it shows scheduling and accounting in a form small enough to hold in my head, before Kubernetes adds its own vocabulary.

What I built

Three Debian 12 VMs on Proxmox: one controller (slurm-ctl) and two compute nodes (lab-node1, lab-node2), 2 vCPUs and 3 GB each, running Slurm 22.05.8. Everything is configured by one idempotent Ansible playbook, so a second run changes nothing.

Slurm is a queue plus a placement engine. You say what a job needs, which is its resource request (CPUs, memory, nodes, time limit), and the controller decides when and where it runs. The other pieces make that work across several machines:

  • munge lets machines authenticate each other.
  • slurmd runs the job.
  • cgroups enforce the limits.
  • NFS makes files visible wherever the job lands.
  • slurmdbd keeps a permanent record.

What runs where in the cluster Diagram: the three VMs and what runs on each.

How it works: one job from start to finish

Life of one Slurm job Diagram: the second compute node is not drawn.

  1. sbatch hello.sbatch talks to slurmctld on slurm-ctl, carrying a munge credential: a signed, timestamped token.
  2. slurmctld records the job as PENDING. Its scheduler looks for nodes with free cores and memory.
  3. It tells slurmd on the chosen node to start the job, again authenticated with munge.
  4. slurmd starts a slurmstepd, puts the job in its own cgroup, and runs the script as your user. That is why user IDs must match on every node.
  5. The script writes its output to /shared, which has the same path on every node because of NFS.
  6. When the job ends, the record goes to slurmdbd and then MariaDB. sacct reads it back from there.

What each piece does, and what breaks without it

PieceWhyIf missing or wrong
munge, shared keyNodes trust each other without passwordsAuth errors; nodes show down
Same uid/gid on every nodeA job runs as the same user everywhereWrong file ownership; munge can fail
chrony (time sync)munge tokens expire quickly”Credential expired” errors
Identical slurm.confOne shared cluster definitionNodes refuse to register
Controller-local StateSaveLocationThe queue survives a restartQueue lost on restart
NFS /sharedJob files at the same path anywhereJobs fail on the wrong node
cgroupsEnforce the CPU and memory requestedA greedy job starves its neighbours
slurmdbd + MariaDBAccountingThe cluster runs, but sacct is empty

The NFS share is for job data, not for Slurm’s own state. The controller keeps that state on its own local disk, in the directory set by StateSaveLocation (/var/spool/slurmctld).

Where state, job files and history live Diagram: three separate stores, and what happens if each one is lost.

What a node checks before it trusts a request

Checks on a munge credential Diagram: the credential, the three checks on the receiving node, and the failure each one produces.

Names and addresses

Why /etc/hosts and NodeAddr are set explicitly Diagram: the default cloud-init behaviour and what the lab does instead.

Running jobs in containers: Pyxis and Enroot

Large GPU clusters rarely run a job directly on the host’s libraries. Users bring their own software stack as a container image, and Slurm has to start it. The usual pairing is Enroot, a tool that turns a container image into an unpacked root filesystem and runs a process in it without a container daemon, and Pyxis, a Slurm plugin that adds container options to srun and sbatch. Pyxis is a SPANK plugin: slurmd loads it from plugstack.conf and calls it as each job step starts, and it asks Enroot to import the image, create the container and run the step inside it.

  • --container-image takes a registry name such as python:3.12-slim or the path to a squashfs file made earlier with enroot import.
  • --container-mounts bind-mounts host paths, such as the shared filesystem, into the container.
  • --container-name keeps the unpacked container on the node so a later step can reuse it.

Nothing about scheduling changes. Slurm still allocates the nodes, the cgroups still enforce the limits, and sacct still records the step. The container is how the process sees its filesystem.

Commands and results

  • A two-node sbatch -N2 job completed (exit code 0:0) on both compute nodes, wrote its output to /shared, and sacct returned the job and its steps. The cgroup paths in the job’s output named the real nodes, which is how I confirmed where it ran.

  • Drain drill: I drained a node while a job was running on it:

    • 04:35:36, a 40-second job is running on lab-node1.
    • 04:35:40, I set the node to drain. State becomes draining.
    • A second job with no node preference ran on lab-node2. A third, pinned to lab-node1, stayed PENDING with reason ReqNodeNotAvail.
    • 04:36:16, the running job finished and the node became drained, with my reason kept.
    • 04:36:19, I resumed the node. At 04:36:21 the pinned job ran.

    So drain lets running work finish and holds new work until the node returns.

  • Unresponsive node drill: I stopped slurmd on lab-node1 and checked sinfo every ten seconds. SlurmdTimeout is 300 seconds on this cluster.

    • For the first 34 seconds the node still showed idle. By 45 seconds it showed idle*: the asterisk means the controller is not getting answers from the node, and the node still counts as usable until the timeout.
    • About 228 seconds after I stopped it, the state became down* with the reason Not responding. That is under the 300 seconds. The slurm.conf documentation says the controller waits SlurmdTimeout seconds for slurmd to respond before it sets the node to DOWN, and that it pings the daemons over a hierarchical communication tree. I read that as a wait counted from the last time the node answered, which was before I stopped the daemon, so the visible time is shorter than the setting.
    • When I started slurmd again the node was idle at the next check, one second later, with no scontrol command, because ReturnToService=2 lets a node that registers with a valid configuration return by itself.

    So down* is a node the controller cannot reach, and the reason field and the slurmd log on that node are the first places to look.

  • Containers with Pyxis and Enroot, installed on both compute nodes. The same two-node step, srun -N2 --container-image=python:3.12-slim hostname, printed both node names, and /etc/os-release inside the step read Debian 13 while the hosts are Debian 12, which shows the step ran in the image. Timings from one run each on 2 vCPU VMs:

    • no container: 2.3 s
    • registry image: 20.3 s on the first run and 27.2 s on the second. Pyxis printed imported docker image on every run, so the registry name is imported again each time, and the layer cache saves only the download.
    • squashfs file on /shared: 3.0 s. The 82 MB file took 7 s to build with enroot import.
    • named container: 4.6 s to create and 0.7 s to start again on the same node.
    • A bind mount of /shared/enroot-demo as /data let the container write a file that appeared on the host with the job user’s ownership. A two-node sbatch script with the squashfs image completed in 1 s in sacct.

    So a container adds seconds if the image is a file on shared storage and tens of seconds if every step pulls from a registry, which is why GPU sites import once and run from a file.

  • Ansible idempotency: two consecutive full runs reported changed=0 on all three hosts.

The drain drill timeline Diagram: the drain drill, using the timestamps from the log.

Challenges

Things that went wrong along the way:

  • A wrong diagnosis of mine. A node refused SSH and I blamed a host-key conflict. It was simply a new host that had not been accepted yet under non-interactive SSH.
  • NFS mount denied. Ansible handlers run at the end of a play, so the export was not reloaded before the client tried to mount. meta: flush_handlers fixed the ordering.
  • A playbook that was not idempotent. Two roles set the same directory to different modes (0750 and 0755), so every run flipped it. I aligned them.
  • /etc/hosts and cloud-init. cloud-init rewrites /etc/hosts on every boot and maps the node’s own name to 127.0.1.1, which confuses Slurm. I turned that off and let Ansible own the file. That fix did not hold through a power cycle. The hypervisor generates the cloud-init data again at power-on and its setting overrode mine, so after a hard power-off a node could not resolve the controller’s name and Slurm kept it DOWN. I found this later, in Part 4, and fixed it by disabling cloud-init on the nodes in the Ansible role. It then held through three power cycles. A fix is not finished until it has been tested across a power loss.
  • Container runtime state needs a directory that exists. Enroot keeps its per-user runtime files under $XDG_RUNTIME_DIR, which only exists for users with a login session. A batch job on a node where nobody is logged in failed with mkdir: cannot create directory '/run/user/1000', and with Pyxis marked required the whole step failed. Setting ENROOT_RUNTIME_PATH in /etc/enroot/enroot.conf to a path under /tmp fixed it. Without that setting a node that works for one person can fail for the next job.
  • A named container lives on one node. --container-name on a second node does not find the container made on the first, so steps that must share a container have to run where it was created.

Differences with a true HPC cluster like Meta’s

This lab is three small VMs, so it says little about how a large cluster behaves. Meta’s published numbers show the gap. In March 2024 Meta described two clusters of 24,576 NVIDIA H100 GPUs each, and its earlier AI Research SuperCluster was planned to connect 16,000 GPUs. The lab has none.

  • Network fabric. My nodes share one virtual network on a home LAN. Meta built one of its clusters on RoCE with Arista 7800 switches and the other on NVIDIA Quantum2 InfiniBand, and says it changed its internal job scheduler to place jobs with network topology awareness. Slurm can do this with a topology.conf file. I haven’t configured it, because my nodes have no switch layout to describe.
  • Storage. My /shared is one NFS export on the controller. Meta describes a flash-optimised distributed store behind a FUSE interface for checkpoints, plus a parallel NFS deployment built with Hammerspace, at exabyte scale.
  • Scheduling policy. I did not try priorities and fair-share or preemption.
  • Controller failure. The lab has one controller. Slurm supports standby controllers through several SlurmctldHost entries, where the first is primary and the rest wait as backups. I haven’t tried that, or what happens when the controller itself fails. I also haven’t tested whether the /etc/hosts fix survives a reboot.
  • Accounting and limits. The accounting database is one MariaDB on the controller with no separate backup. I haven’t set quotas, and I have no monitoring beyond reading sinfo and logs.
  • Experiments not yet run. Stopping munge, stopping slurmd and unmounting /shared are written down as predictions, and I haven’t run them.
  • Containers at scale. I ran Pyxis and Enroot on two nodes with an 82 MB image and no GPU. A GPU cluster adds the NVIDIA container hooks that expose the GPUs inside the container, images of many gigabytes, and a parallel filesystem to hold the squashfs files, none of which I have here.
  • Slurm and Kubernetes together. Some sites queue batch work on Kubernetes with Kueue, which admits jobs against quotas, and the Slinky project runs Slurm itself on Kubernetes. I ran neither, and the two schedulers in this lab do not know about each other.

Meta’s post describes an internal scheduler, not Slurm, so this comparison is about scale and design and not about one tool.

FAQ

What is chrony?

chrony is an implementation of NTP, the Network Time Protocol, which keeps a machine’s clock in step with time servers (chrony). The cluster needs it because munge credentials carry a timestamp and expire after a time-to-live, 5 minutes by default. The fastest and slowest clocks in the cluster have to be within that window. Slurm’s quick start says the same: munge requires synchronized clocks, usually through NTP. In the lab, Ansible installs and starts chrony on all three VMs.

What is cloud-init?

cloud-init is the standard tool for first-boot setup of a cloud image. On boot it reads data supplied by the platform, such as the hostname, network settings and SSH keys, and applies it. My VMs use the Debian 12 cloud image, and the creation script passes each VM’s static address and first SSH key through cloud-init. Its side effect is the one in the Challenges list: it rewrites /etc/hosts on every boot and maps the node’s own name to 127.0.1.1, which confused Slurm. I first turned that off with manage_etc_hosts: false, which did not survive a power cycle (see Challenges), and now disable cloud-init on the nodes and let Ansible own the file.

What is saved in slurm.conf, and must the hardware be the same?

slurm.conf is the cluster definition, and it is identical on every node. In this lab it holds the cluster name and controller host, the authentication type (munge), the process-tracking and task plugins (cgroup), the state directory, the scheduler (backfill) with cores and memory as consumable resources, the accounting settings (slurmdbd), one line per compute node, and the partition.

The hardware does not have to match. The file is the same everywhere, but each NodeName line describes that node’s own CPUs and memory, so a cluster can mix node types, grouped into partitions or labelled with features and GPUs. My two nodes are identical because my template gives every compute node the same values. What a node must do is register with at least the resources the file says it has. If it registers with fewer, Slurm marks it INVAL and it takes no work until slurmd is restarted with a valid registration.

Which node states exist besides draining?

These come from the sinfo documentation, which covers a newer Slurm than my 22.05.8, so a few of them may not exist in my version. In my drills I only saw idle, draining, drained, idle* and down*.

GroupStatesMeaning
Working normallyIDLE, ALLOCATED, MIXED, COMPLETINGfree, fully used, partly used, or finishing the jobs it had
UnavailableDOWN, DRAINED, DRAINING, FAIL, FAILING, INVAL, UNKNOWN, FUTUREfailed or unreachable, removed by an administrator, expected to fail, registered wrongly, not yet checked by a restarted controller, or not yet configured
Reserved or plannedRESERVED, MAINT, PLANNED, BLOCKEDin a reservation, in a maintenance reservation, set aside by the backfill scheduler for a higher-priority job, or blocked by an exclusive topology job
PowerPOWER_DOWN, POWERING_DOWN, POWERED_DOWN, POWERING_UPpowering off, off, or powering on
RebootREBOOT_REQUESTED, REBOOT_ISSUEDa reboot asked for, or sent to the agent

A suffix can follow a state: * means the node is not responding (the unresponsive node drill above shows it appearing on idle first and on down after SlurmdTimeout), ~ means it is powered off, # means it is powering up or being configured, and $ means it is in a maintenance reservation.

Next

Take the quiz to test your understanding: Slurm quiz

Part 2: the same machines, running Kubernetes.

Comments