Kubernetes on the same three VMs

Goal

Part 1 was a scheduler for batch jobs. This part is the other kind of scheduler: Kubernetes keeps long-running services at a desired state and moves them when a node fails. I wanted to see what happens to a service when a node goes away, how an operator takes a node out for maintenance, how a cluster is upgraded one minor version, and how autoscaling reacts to load. I also wanted to see what it is like to run both systems on the same machines.

Terms used in this post

Kubernetes has a lot of vocabulary. These are the words I use below, in the order they build on each other.

  • Node: one machine in the cluster. Here, one VM.
  • Container: a packaged program that runs in isolation on a node.
  • Pod: the smallest thing Kubernetes runs. It wraps one container (sometimes a few) and gets its own network address.
  • Deployment: a rule such as “keep three copies of this pod running”. Kubernetes creates or removes pods until the rule is true.
  • Replica: one of those copies. “Three replicas” means three identical pods.
  • Service: one stable address in front of a set of pods. Pods come and go and their addresses change, so clients talk to the Service.
  • Control plane: the parts that decide what should run where. The API server stores the desired state and the scheduler picks a node for each new pod.
  • Kubelet: the agent on each node that starts the containers and reports back.
  • Evict: remove a pod from a node. If a Deployment owns the pod, a replacement is created somewhere else.
  • Cordon: mark a node so no new pods are placed on it. Pods already there keep running.
  • Drain: cordon a node, then evict the pods on it, usually before maintenance.
  • Taint: a marker on a node that means “do not place pods here” or “do not keep pods here”.
  • Toleration: a setting on a pod that says it can put up with a taint, and for how long.
  • PodDisruptionBudget: a limit on how many replicas may be down at once during planned work such as a drain.
  • HorizontalPodAutoscaler: a rule that adds or removes replicas based on load, within a minimum and a maximum.

How the terms fit together Diagram: a Deployment, the control plane, a Service, two nodes with their pods, and a taint.

What I built

I installed k3s on the three Debian VMs from Part 1: the server on slurm-ctl and an agent on each of lab-node1 and lab-node2. Slurm from Part 1 is still on these machines. I come back to that at the end. Two new Ansible roles, k3s_server and k3s_agent, do the install and join the agents with a token read from the server. Before touching anything I took a Proxmox snapshot of each VM so a bad install could be rolled back.

The k3s cluster Diagram: the three VMs and what k3s runs on each.

The test workload is a small web Deployment (nginx, three replicas, a CPU and memory request on each pod) behind a ClusterIP Service. Everything else in this post is something I did to that Deployment or to the nodes under it.

How it works

Kubernetes has a control plane and node agents, the same split as Slurm’s controller and slurmd.

  1. The API server stores the desired state, for example “three copies of web”.
  2. A controller notices when fewer than three pods exist and creates more. The scheduler picks a node for each new pod, based on the CPU and memory the pod asks for.
  3. The kubelet on that node starts the container and reports back.
  4. A Service gives the pods one address and sends traffic to the ready ones.

When a node goes silent, this is what happens:

  1. After a grace period, 40 seconds by default, the control plane marks the node NotReady.
  2. It puts a taint on the node. The taint is a marker that says pods should not stay there.
  3. Every pod has a toleration for that taint, which says how long the pod will wait before it gives up the node. The default is 300 seconds.
  4. When that time is up the pod is evicted, and the Deployment creates a replacement on a healthy node.

So two settings decide most of what an operator sees: the 40 second grace period and the 300 second toleration. I measured both.

Commands and results

Deployment and Service. Three replicas started on three different nodes in about four seconds once the image was pulled. I scaled to five and then six and the pods spread across all three nodes. The Service answered with HTTP 200 from the controller and from a compute node.

Node failure. I stopped the k3s agent on lab-node2. The containers on the node kept running, so the control plane was reacting to silence, which is what a network partition looks like.

  • With the defaults, the node went NotReady at 44 seconds. The pod on it was deleted at about 344 seconds, and a replacement was running and ready on another node at 345 seconds.
  • 344 is 44 plus 300. The toleration timer starts when the node is tainted, not when it fails.
  • I then gave the Deployment a toleration of 30 seconds for both taints. The node went NotReady at 46 seconds and the replacement was ready at 76 seconds.

Node failure drill timelines Diagram: the two drills. Each has its own time axis.

I did not cut the power to a VM, so I have no numbers for a node whose containers stop as well.

Drain with a PodDisruptionBudget. A budget says how many replicas must stay available during voluntary disruptions such as a drain.

  • With minAvailable: 3 on a three-replica Deployment, kubectl drain lab-node1 evicted nothing and gave up after its 40 second timeout. The node was left cordoned.
  • With minAvailable: 2, the same drain finished in about a second and the evicted pod was replaced elsewhere.

This is the same shape as the Slurm drain in Part 1, except that Kubernetes moves the work and Slurm waits for it to finish.

Minor upgrade. I upgraded k3s from 1.36.6 to 1.37.2 with a playbook: server first, then each agent in turn with drain, install, wait for Ready, and uncordon. A probe hit the Service from the controller once a second. Three probes failed in a row (about six seconds) while the server restarted. None of the 29 probes failed during the agent upgrades. All nodes ended on 1.37.2 and Slurm stayed healthy.

Removing and adding an agent. I drained lab-node2, deleted the node, ran the uninstall script, and re-ran the install playbook. The node was Ready about 14 seconds after the playbook started. A scale to six replicas then spread two pods per node.

Autoscaling. A HorizontalPodAutoscaler targeted 50% of the pods’ CPU request, with a minimum of three and a maximum of eight replicas. Three small pods looping wget against the Service provided the load.

  • The Deployment went from three to six and then to eight replicas within about a minute.
  • At eight replicas the average was still 59% against the 50% target, so the maximum was what stopped it.
  • After I removed the load the Deployment stayed at eight for about five and a half minutes, then went to four and then three. That is the default five minute scale-down stabilization window plus metrics delay.

Taints and tolerations. I labelled lab-node2 and tainted it NoSchedule. A pod with a node selector and no toleration stayed Pending with the reason “untolerated taint”. The same pod with a toleration ran on the node. The web pod that was already there was not evicted, because NoSchedule only affects new placement. This is a scheduling control on CPU nodes with an invented label, not GPU scheduling.

Requests and allocatable. Each node reports 2 CPUs and about 2984 MB as allocatable, the same as its capacity, because k3s reserves nothing by default. Slurm is told 2800 MB on the same machine. Neither scheduler knows about the other. I also checked that a Slurm job asking for 64 MB and allocating 300 MB was still killed by its cgroup, so the memory limit holds next to the kubelet.

Challenges

These are the problems that cost me time, and each one is something a fleet operator meets too.

  • Upgrade automation that damaged the node it was upgrading. On lab-node1 the k3s installer rewrote the agent’s environment file from its own environment. I had not passed the server URL and token again, so the file ended up empty and the agent looped on --server is required. The node stayed cordoned on the old version until I fixed the playbook to pass both values. The pods kept running the whole time, so it was a repair and not an outage. In a fleet the same thing is a rolling upgrade that stops partway with one node half done, which is why upgrades go one node at a time with a check between nodes.
  • Configuration drift between the provisioning source and the running fleet. After the upgrade my pinned version still said 1.36.6. When I removed and re-added lab-node2, it came back one minor version behind the rest. The same thing happens when a machine is reimaged after a hardware swap from an out-of-date build definition. The pin has to move with the upgrade.
  • A drain that waits forever. A disruption budget that allows zero disruptions makes a drain hang until its timeout, and the node is left cordoned. During a maintenance window that means a stalled rollout, or a node that is cordoned and forgotten.
  • Failure detection is slow by default. With the defaults, the service ran at reduced capacity for almost six minutes after a node went silent. I could cut it to about 76 seconds, at the cost of evicting pods during short network blips. Choosing that value is a real operations decision.
  • Two schedulers on the same hardware. Each one believes it owns all of a node’s memory, as the diagram in the comparison section shows. Clusters that share nodes between batch and service workloads run into this. The controller VM went from about 400 MB used to about 1.2 GB used once k3s was installed, and its available memory dropped to about 1.6 GB while eight web pods ran. Slurm still enforced its cgroup limit next to the kubelet, but neither scheduler counts what the other has promised.

What this lab cannot show is how real hardware fails. I stopped a software agent. I did not see a disk slowly failing, a flapping network port, a node that is half alive, or several machines failing together.

Differences with a production Kubernetes cluster

This is three small VMs with one control-plane node, so it says little about how a large cluster behaves. Kubernetes documents support for clusters of up to 5,000 nodes with its own set of limits and tuning.

  • Control plane. I have one server with the default embedded datastore. k3s can run several servers with embedded etcd. I tried that afterwards, and what happens when servers are lost, in Kubernetes 2.1.
  • Upgrades. I went one minor version, server first, with Kubernetes’ version skew rules in mind. I did not run the cluster through several versions or test a rollback.
  • Node autoscaling. The HorizontalPodAutoscaler only changes pod counts. Adding and removing machines needs a cluster autoscaler and a cloud or a machine pool behind it.
  • Reserved resources. Real clusters reserve CPU and memory for the system and the kubelet. I left the defaults, and the memory pressure in the Challenges section follows from that.
  • Disruptions. I only used the PodDisruptionBudget and the taint-based eviction timers. I did not test node pressure eviction or a rolling update against a budget.
  • Monitoring. I looked at kubectl top and kubectl describe node in this post. Prometheus and Grafana come in the next part.
  • Managed control planes. EKS Auto Mode, GKE and AKS run the control plane for you and automate much of the node upgrading that I did by hand here.
  • Node provisioning. Karpenter creates nodes directly in response to pods that cannot be scheduled, which is a different approach from the cluster autoscaler above. I did not use either.
  • Delivery. I applied changes with kubectl and Ansible. Production teams usually deliver from a git repository with a pull-based tool such as Argo CD or Flux, so the cluster follows the repository.
  • Networking and policy. I used the networking k3s ships with and wrote no network or admission policy. Production clusters often use the Gateway API for incoming traffic, network policies with a network plugin such as Cilium, and admission rules from Kyverno or Gatekeeper.

Slurm and Kubernetes side by side

Part 1 and this part run on the same three VMs. Slurm and k3s are installed on every machine, and both schedulers see all of each machine’s memory.

Slurm and k3s on the same three VMs Diagram: both stacks on each VM. The captions give the memory each scheduler believes it has.

The two systems answer different questions. Slurm decides when and where a finite job runs and how long it may take. Kubernetes keeps services running and replaces them when something fails.

SlurmKubernetes (k3s)
Unit of workA job: a script with a request for CPUs, memory, nodes and timeA pod, kept alive by a Deployment
Does it end?Yes. A job finishes or hits its time limitNo. A service is meant to keep running
What you hand itA job request; the scheduler may queue itA desired state, such as three replicas
Controllerslurmctld, with state on its own diskAPI server and scheduler, with state in the datastore
Node agentslurmdkubelet
Taking a node outDrain: running jobs finish, nothing new startsCordon and drain: pods are evicted and replaced elsewhere, if the disruption budget allows
A node goes silentThe controller marks it down after SlurmdTimeout (300 seconds by default). I did not run this one in SlurmNotReady at about 44 seconds, replacement running at 345 seconds with the defaults
Historyslurmdbd and MariaDB, read with sacctNo long-term record built in. Metrics need something like Prometheus
Limitscgroups enforce the CPU and memory a job asked forRequests are used for placement, limits enforced with cgroups

The memory numbers in the diagram are the problem with sharing nodes. Each scheduler counts the whole node and neither knows what the other has promised. Slurm still killed a job that went over its limit while the kubelet was running, so the enforcement held. The accounting did not.

FAQ

What is k3s?

A lightweight Kubernetes distribution packaged as one binary, with a bundled ingress controller, a metrics server and a local storage provisioner. It is the same Kubernetes API, with fewer moving parts to install.

What is the difference between cordon and drain?

Cordon marks a node unschedulable, so no new pods land there. Drain cordons it and then evicts the pods already running on it, respecting any disruption budgets.

What does a PodDisruptionBudget protect against?

Voluntary disruptions: drains, upgrades and similar operator actions. It does not protect against a node crashing.

Why did the new pod appear only after about 345 seconds?

Pods have a default toleration of 300 seconds for the not-ready and unreachable taints, and the timer starts once the node is tainted. The taint came about 44 seconds after the agent stopped.

Next

Take the quiz to test your understanding: Kubernetes quiz

Part 4: a model-serving workload on this cluster, with load tests and a proper monitoring stack.

Side track: the Advanced Kubernetes track goes deeper on this cluster, starting with Kubernetes 2.1: three control-plane servers and etcd on k3s. The homelab series carries on with Part 4 above.

Comments