Goal
In the Kubernetes post the cluster had one server and two agents. The server held all of the cluster’s state in a single SQLite file, so losing that machine meant losing the control plane. This post changes that. I wanted to move to three servers that share the state through etcd, then see what the API does when one server is gone, what it does when two are gone, how a snapshot is taken and restored, and how servers are restarted one at a time.
Terms used in this post
- Control plane: the API server, the scheduler and the controllers. They decide what should run. In k3s all of them run inside one
k3sprocess on a server node. - Server and agent: a k3s server runs the control plane and can also run pods. An agent only runs pods.
- Datastore: where the cluster state lives. The API server is the only program that talks to it.
- SQLite: a single-file database. It is k3s’s default datastore for one server.
- etcd: a replicated key-value store. Kubernetes uses it for all cluster state.
- Member: one copy of etcd. In k3s each server runs one member.
- Quorum: a majority of members. etcd accepts a write only when a majority of members has it. With three members the majority is two.
- Snapshot: a point-in-time copy of the etcd data in one file.
- Cluster reset: starting etcd again from a snapshot, as a cluster of one member, so the others can join it.
Diagram: all three servers up, one down and two down, with the majority rule for each.
What I built
I started from the three VMs and the k3s cluster of the previous post, and took a Proxmox snapshot of each VM first. By then the cluster also ran Argo CD, Kyverno, Kueue, KEDA, Crossplane, NetBox and a few other workloads from later parts, 34 deployments in all.
The change happened in three steps:
slurm-ctlwas restarted with--cluster-init. k3s converts the SQLite database into an etcd data directory on that start.lab-node1and thenlab-node2were changed from agents to servers that join the first one, using an Ansible playbook,k3s-ha.yml.- Three small scripts did the drills below: one for quorum, one for snapshot and restore, and one for a rolling restart.
All three nodes now report the roles control-plane,etcd.
How it works
Every write to Kubernetes goes through the API server to etcd. etcd keeps copies on all members and agrees on each write by majority vote (it uses the Raft protocol). A cluster of three members can lose one and keep going. A cluster of five can lose two. The k3s documentation asks for an odd number of servers because adding a fourth member does not let the cluster survive one more failure. It only raises the number needed for a majority.
Pods already running on a node do not depend on the control plane while it is away. The kubelet on each node keeps its containers running, and the network rules that make a Service work stay in place. What stops is everything that needs a decision or a write: new pods, scheduling, scaling and any kubectl command.
A snapshot is a copy of the etcd data. To restore one, you stop the servers, start the first one with a reset flag that points at the snapshot file, and let the other servers rejoin from nothing.
Commands and results
Migration. I re-ran the k3s installer on slurm-ctl with --cluster-init. The SQLite file was 41 MB. A probe against the API once a second saw it unreachable from 3.2 to 15.8 seconds after the start of the change, about 14 seconds. k3s left the old file next to the new data as state.db.migrated. All 34 deployments were still there afterwards. The server process also went from 2.3 GB to 1.56 GB of resident memory. K3s’s resource profiling lists nearly the same minimum memory for the SQLite and embedded etcd backends (1,596 M and 1,606 M on one test machine), so the backend is unlikely to be the reason. The migration also restarted the k3s process, and a fresh process starts without the memory that the old one had accumulated (my inference).
Joining two servers. The playbook changed lab-node1 first, and lab-node2 after it. During both joins I probed the first server’s API once a second, 486 probes in all, and none failed. Each node showed control-plane,etcd and Ready a few minutes after the start of its change.
One server down. I stopped k3s on lab-node2.
kubectl get nodesanswered in 0.5 seconds and creating a ConfigMap took 0.4 seconds.- The
webService still answered with HTTP 200. - About 55 seconds later the node showed
NotReadyand the other two showedReady.
Two servers down. I then stopped k3s on lab-node1 as well, so only slurm-ctl was left, one member out of three.
- About 12 seconds later the k3s process on
slurm-ctlexited with status 1. systemd started it again, and while the majority was missing port 6443 refused connections. kubectlcommands, reads and writes, all failed with connection refused.- The journal showed “failed to publish local member to cluster through raft” and “Failed to test etcd connection”.
- The
webService still answered with HTTP 200, so the pods kept running. - After I started
lab-node1again, which made it two members out of three, the first write succeeded 19 seconds after the start command. When I startedlab-node2too, all three nodes wereReadyin about a minute.
Image: the results of stopping one server and then two. The text source is kept with the lab files.
I did not test whether a pod that crashes during the outage gets replaced. That needs the controllers to write to the API, which was refused, but I only checked the Service.
Snapshot and restore. I saved a snapshot with k3s etcd-snapshot save. It took 0.7 seconds and the file was 25.6 MB. After that I created a ConfigMap and deleted the gw-demo namespace, which were the changes to undo. Then I stopped k3s on all three servers and ran k3s server --cluster-reset --cluster-reset-restore-path=<snapshot> on slurm-ctl. The reset itself took 3.1 seconds. After I started k3s again the API answered 13 seconds after the reset began. I removed the etcd data directory on the other two servers and started them, and all three nodes were Ready 61 seconds after the reset began.
The result matched the snapshot. The namespace was back with its four pods running, and the ConfigMap I created after the snapshot was gone.
Image: the snapshot and restore run, with the three-node state at the end.
Rolling restart. I restarted the k3s service on each server in turn, lab-node2, lab-node1 and slurm-ctl last, and probed all three API endpoints once a second. Each endpoint was unreachable for 3 seconds while its own server restarted. The other two kept answering. I could not run a version upgrade, because 1.37.2 is already the newest k3s release on every channel. A rolling restart is the same order an upgrade uses, and I wrote an upgrade playbook for three servers, k3s-upgrade-ha.yml, that I have only syntax-checked.
Image: the rolling restart, with the unreachable time for each server’s API endpoint.
Challenges
- A drain that could not proceed. Draining
lab-node2stalled. Thewebbudget requires two replicas and showed zero allowed disruptions, because onewebpod from the earlier drain was left in theCompletedstate and so was not ready. A budget counts ready pods, not pods that exist. I deleted the finished pod and the drain could continue. - Local volumes tie a pod to a node. A database that uses a local-path volume lives on one node, and the agent uninstall script removes the whole
/var/lib/rancherdirectory, which includes those volumes. Removing or rebuilding a node that way loses its local data. A node that holds such volumes has to be converted in place, with the agent service stopped and the server installed over it, or its data has to be moved first. - Node-local images disappear with the node’s data. The same uninstall removes container images that were loaded onto the node directly. A Deployment that never pulls then cannot start a pod there. An in-cluster registry makes images reachable from any node, so replacing a node does not strand a workload.
- Quorum loss looks like a dead API, not a read-only one. I had expected a cluster without a majority to at least answer reads. Here the API stopped answering altogether, which is a different situation to plan and alert for.
What this lab cannot show is how members fail on real hardware: a slow disk making etcd miss its heartbeat, a flapping network link between servers, or a rack that loses power. I stopped a service. In the future I’m planning to purchase a hardware kit to help practice further.
Differences with a production setup
- Topology. I run etcd on the same machines as the API servers, which Kubernetes calls the stacked topology. Production setups often use it too, and some use an external etcd cluster so the two can be sized and failed separately.
- Member count and failure domains. I have three members on three VMs on one hypervisor host. Production setups spread three or five members across racks or zones, so one failure takes out one member.
- Fixed address. I joined servers to the first server’s address, and my kubeconfig points at it. Production setups put a fixed registration address such as a load balancer or virtual address in front of the servers, so the cluster has no single entry point.
- Dedicated control-plane nodes. My servers still run workloads. Production setups usually keep control-plane nodes free of them.
- Disk. etcd is sensitive to disk latency. The k3s documentation warns about slow disks. I did not measure my disks.
- Backups. I took one snapshot by hand and restored it on the same machine. Production setups take scheduled snapshots and keep them off the cluster, for example in object storage, and rehearse the restore. Kubernetes describes operating etcd clusters including its own backup steps.
- Changing membership. I did not remove or replace a member. etcd documents runtime reconfiguration and its failure modes.
- Upgrades. I did not run a version upgrade of three servers. Production setups follow the version skew policy, upgrade one control-plane node at a time, and many drive the whole sequence with Cluster API. k3s describes the manual procedure.
- Draining. Production setups drain with a disruption budget that matches the workload, as in Safely Drain a Node.
FAQ
Why three servers and not two?
Two members need both to be up for a majority, so losing either one stops writes. That is worse than one server, because there are now two machines that can fail. Three members can lose one.
What happens to my pods when the API is unreachable?
The ones already running keep running. In the drill the Service still answered. Nothing can be scheduled, scaled or changed until a majority of members is back.
Is a snapshot a backup?
It is the file a backup is made from. A copy that sits on the same machine as the cluster does not survive losing that machine.
Why did the restore need the other servers’ data removed?
They still held etcd data from the state after the snapshot. Removing it let them join the restored member as new members and copy its data.
Repeating with real hardware
I would use three physical servers with separate disks, and check the disk’s write latency before trusting it with etcd. I would pull the power on one server, then on two, and time the recovery. I would also take a snapshot to another machine and restore it there, which is the case that matters when a whole server is lost.
Next
Take the quiz to test your understanding: Kubernetes quiz
Replacing the default network plugin with Cilium and writing network policy is planned for the next post in this track.
Comments