Learning infrastructure by building it: series introduction

Goal

I want to understand how large compute infrastructure is run, and reading is not getting me there. I’ll be documenting my journey to break this into a series of frameworks and skills in my learning journey. The initial study will be done virtually until I reach a point where I’m blocked by hardware.

The lab

Three Debian 12 virtual machines (2 vCPUs each) on a single Proxmox host, built from scripts and configured with Ansible. There are no GPUs in it.

That is where the series starts, and the lab grows as it goes. Part 1 puts a Slurm scheduler on the three machines, and Part 2 adds a Kubernetes cluster on the same three. Part 3 is reading on how industry uses the two schedulers, and Part 4 puts a model server, object storage, Ray and Terraform on top of it, and Part 5 adds the tools a shared cluster needs: job queues, event-driven scaling, delivery from git, policy, a gateway and self-service provisioning. In Part 6 the lab gets a few extra short-lived machines around it, for emulated management controllers, an inventory, a network boot server, RAID drills and a separate management network. Part 7 adds an emulated cloud account and emulated GPUs, and a virtual NUMA machine. Part 8 is reading only, on the GPU hardware the lab cannot run. The three machines stay the core of it.

Diagram: the nine parts and where each stands today. Each part and each post is a link.

What I want to learn in each part

The map above is for finding posts. This table says what I was trying to learn from each part and what it costs.

#PartWhat I want to learnCost
0FoundationsHow the lab is put together, why LXC containers and Tailscale fit, and how Go and Python differ for toolingfree
1SlurmHow a scheduler places work, tracks nodes and keeps accountingfree
2KubernetesThe cluster lifecycle: upgrades, adding and removing nodes, autoscaling, capacity and monitoringfree
3Slurm and Kubernetes in industryWhich scheduler large GPU sites run, why many run both, and what changes for rack-scale NVLink systems, from public evidencefree, reading only
4ML platform on KubernetesWhat it takes to serve a model and move data to it, with Ray, infrastructure as code and a command line toolfree
5Platform layer on KubernetesHow a shared cluster is governed: delivery from git, queues and quotas, scaling, policy, traffic and self-servicefree
6Bare-metal fleet, emulatedHow a fleet of servers is brought up and kept healthy without real hardware: management controllers, power, inventory, hardening, provisioning, storage and networksfree
7Cloud and GPUHow cloud and GPU capacity is requested, watched and scheduled, tried on emulators first so nothing is spent, plus how memory topology affects placementthe real run spends money
8ReferencesHow GPUs connect inside a server and between servers, what the platform names mean, and what GPU racks need for power and coolingfree, reading only

Part 7 holds the only real run that costs money. Part 8 has no lab: it is reading and arithmetic on hardware I do not have, so each post says that at the start.

Challenges

Some of what I want to learn can’t be done in this lab. I have no real GPUs or rack, no out-of-band management controllers, no high-speed network fabric or power distribution gear, and no production traffic or other people depending on it.

Where a part touches one of them, I’ll use an emulator or a worked exercise on paper and say which in the post. In the future I’ll detail the hardware kit I’ll buy to help practice further.

Next

Take the quiz to test your understanding: final quiz for Parts 1 to 8

Part 0: the foundations, Part 1: building a three-node Slurm cluster, Part 2: Kubernetes on the same three VMs, Part 3: Slurm and Kubernetes in industry and Part 4: a model server on Kubernetes under load.

Comments