Goal
Almost everything in this series used one of two languages. The tools that run a cluster (k3s, Prometheus, Terraform, MinIO) are written in Go, and most of the work I ran on the cluster (the model server, the benchmarks, the automation glue) I wrote in Python. Engineers who work on infrastructure are often asked why, and the good answers are about concepts such as typing, concurrency, memory management and deployment, not about syntax.
This post first sets out the differences between the two languages in those terms. After that comes a case study: one small CPU-bound pipeline written in both languages and timed on a lab VM, so the concepts have a measurement next to them. It sits next to the Proxmox and Tailscale post in Part 0. The lab has no GPUs and the pipeline is small, so the numbers describe a 2 vCPU VM and nothing larger.
Terms used in this post
- Compiled language: source is turned into machine code before it runs. Go is one.
- Interpreted language: a program called an interpreter runs the source. Python is one. The standard interpreter, CPython, first compiles source to bytecode and then runs that on a virtual machine.
- Static typing: types are checked before the program runs. Dynamic typing: types are checked while it runs.
- Garbage collector (GC): the part of the runtime that frees memory the program no longer uses.
- Concurrency: several tasks making progress in overlapping time. Parallelism: several tasks running at the same instant on different cores. A program can have the first without the second.
- I/O-bound work spends its time waiting for disks or networks. CPU-bound work spends it computing.
- Goroutine: Go’s lightweight unit of concurrent work. Thousands can run at once.
- GIL (global interpreter lock): a lock in the standard Python interpreter that lets only one thread run Python code at a time.
- Static binary: one executable file with its libraries built in, so the machine needs nothing else installed.
- Resident memory (RSS): the memory a process holds in RAM.
Go and Python: the differences that matter in infrastructure work
| Concept | Go | Python |
|---|---|---|
| Execution | Compiled ahead of time to a native executable | Source compiled to bytecode and run by the interpreter |
| Typing | Static, checked at compile time | Dynamic; type hints are optional and checked by separate tools, not at run time |
| Abstraction | Interfaces are satisfied implicitly by any type with the right methods | Duck typing: an object works if it has the attribute you use |
| Concurrency | Goroutines and channels, scheduled by the runtime onto operating system threads | Threads, asyncio (cooperative, one thread) and processes |
| Parallel CPU work | Goroutines use all cores by default | The GIL means threads do not run Python code in parallel; processes do |
| Memory management | Concurrent tracing garbage collector | Reference counting plus a cycle collector |
| Errors | Returned as values, checked explicitly (Go’s approach) | Exceptions |
| Deployment | go build makes one binary; cross-compile by setting the target operating system and architecture | An interpreter at the right version, plus packages in a virtual environment or an image |
| Tooling | go test, go vet, gofmt and pprof are in the standard toolchain | pytest and the profilers come from the ecosystem or the standard library |
Compilation, typing and what each catches
A compiler rejects a type error or a misspelled field before anything ships. In Python the same mistake surfaces when that line runs, which may be in production at 3 a.m. Type hints and a checker narrow that gap, but nothing forces them. The trade-off is speed of writing: dynamic typing is quicker for a script that runs once.
Concurrency and parallelism
Concurrency is not parallelism. For I/O-bound work, many waiting tasks, Python’s threads and asyncio work well because waiting releases the GIL or yields control. For CPU-bound work in pure Python, one process uses one core, so parallel speed-up needs multiple processes, each with its own interpreter and memory. Go’s goroutines are cheap and use all cores by default, so a control plane that holds thousands of connections and computes between them has less to arrange. The standard interpreter has an optional free-threaded build that removes the GIL, but it is not the default.
Memory
Both languages free memory for you. CPython frees most objects the moment their reference count reaches zero and runs a separate collector for reference cycles. Go’s collector runs alongside the program and is tuned to keep pauses short. The question behind this is usually what you do when memory climbs: look for a leak with a profiler, and in Go use pprof.
Deployment and dependencies
This is the difference that matters most in fleet work. A static Go binary needs only a compatible operating system and architecture. A Python program needs a compatible interpreter version and its packages, some of which contain compiled code for one platform. That is why Python services usually ship as container images, and why a mismatch of interpreter versions between machines is a real source of failures.
Choosing
Go tends to fit tools that run on every node or talk to the Kubernetes API: agents, operators, command line tools and exporters. Python tends to fit the layer where people iterate and where the libraries are: ML code, benchmarks, bring-up scripts and glue. Many stacks use both, split along that boundary. The Kubernetes client library and its operator framework are Go, and ML frameworks are Python-first.
What I built
The rest of this post is a case study: the same pipeline in both languages, to put a measurement next to the concepts above. I wrote two versions of the log pipeline from the Ray post. Each builds 400,000 synthetic web server log lines, parses every line with a regular expression, and adds up counts by status and path and the bytes sent. A run does 48 of those partitions, 19.2 million lines.
- Python: the same script I ran under Ray, with its standard
remodule, run two ways here: one process, and a pool of two processes with multiprocessing. - Go: a short program with the standard
regexppackage, run in one goroutine and with two worker goroutines.
I built the Go program on another machine as a static binary and copied the one file to lab-node1, which has Debian’s own Python 3.11. Nothing else was installed on the VM for either version. The VM has 2 vCPUs, and the other lab workloads were idle.
Diagram: the tools that run the cluster are Go programs, and most of what I ran on it was Python.
How it works
Both programs split the work into 48 independent partitions, so they can run in parallel without sharing anything. The Python pool starts two worker processes, because one Python interpreter runs Python code on one core at a time. The Go program starts two goroutines that take partitions from a channel, and the Go runtime spreads them over the cores.
Each language uses its own random number generator, so the byte totals differ between the two versions (480,979,059,362 and 480,892,571,857). Both processed 19,200,000 lines.
Commands and results
I ran each setup three times and report every run.
| Setup | Runs (s) | Mean (s) | Speed over single-process Python |
|---|---|---|---|
| Python, one process | 53.4, 53.6, 53.5 | 53.5 | 1.0 |
| Python, pool of two processes | 28.0, 28.5, 29.8 | 28.8 | 1.9 |
| Go, one goroutine | 30.7, 30.9, 31.1 | 30.9 | 1.7 |
| Go, two goroutines | 17.0, 17.0, 16.9 | 17.0 | 3.1 |
Python’s regular expressions run in C inside the interpreter, so on this work Go was 1.7 times faster, not the order of magnitude that is sometimes quoted for loops written in pure Python. I did not test a pure Python loop, so I cannot say how that would compare. The two-process pool made Python 1.9 times faster, which is the GIL point from the table: separate processes used both cores.
| Measure | Python | Go |
|---|---|---|
| What I copied to the VM | a 2.5 KB script, which needs the interpreter on the machine | one file of 1.9 MB with symbols stripped |
| Start-up, 200 launches of a trivial program, average | 8.0 ms (python3 -c pass) | 1.4 ms |
| Peak resident memory of a run | 9 MB, and 12 to 13 MB for the pool | 9 to 10 MB |
The memory figures are small for both, because each partition is processed and summarised before the next. The start-up difference matters when a program is launched thousands of times, as with a node agent or a health check, and not for a job that runs for a minute.
For scale, the Ray image I pulled in the previous post was about 930 MB, and the model server image was 338 MB. A Go command line tool is one file of a few megabytes.
Challenges
- Runtime versions are part of the fleet. The VM has Python 3.11.2, the machine I work from has 3.12, and the Ray image I used ships Python 3.11. A Python program that works on one of them can fail on another, and Ray expects the same Python minor version on every node. I did not test a mismatch. The Go binary needs nothing from the node except the right operating system and architecture.
- Python needs a way to ship its dependencies. For the model server I had to build a container image with the right libraries, because the VMs have none of them. That is a packaging and patching job for every machine or image that runs Python code.
- One binary does not remove other problems. A static Go binary still has to be built for each architecture, signed, versioned and rolled out. Here the VMs and my build machine are both amd64, so I did not exercise the architecture side.
What this lab cannot show is a large codebase, long-running services under load in either language, or GPU workloads.
Differences with a production setup
- Scale. One pipeline on one VM does not show how either language behaves in a service with thousands of connections, which is where Go’s concurrency model is usually argued from.
- Languages in the hot path. In a real ML stack the heavy math runs in C++ or on the GPU and Python is the glue, so a Python-versus-Go speed comparison of a regular expression says little about it.
- Free-threaded Python. I used processes for parallel work and did not try the free-threaded build.
- Build and release. A real fleet builds binaries and images in a pipeline for each architecture, signs them, and keeps old versions available. I built one binary by hand.
FAQ
Why is Go common for cluster tooling?
It compiles to one static file with no runtime to install, it has concurrency built in, and the Kubernetes client library and operator tooling are written in it. I saw the first point directly: copying one 1.9 MB file to a VM was all the deployment needed.
Why is Python common for ML and for scripts?
The machine learning libraries are written for it, and it is quick to write and change. Most of what I wrote in Parts 2 and 3 was Python for that reason. In an ML stack Python is usually the glue, and the heavy math runs in C++ or on the GPU.
What is the GIL, and does it stop Python using two cores?
In the standard interpreter it lets one thread run Python code at a time. Separate processes each have their own interpreter, so a pool of two processes did use both cores here and ran 1.9 times faster than one. For waiting on the network, threads and asyncio are fine, because waiting does not hold the lock.
Is Go always faster than Python?
No. On this workload it was 1.7 times faster for one process, because the regular expression engine in Python is native code. The gap is larger for loops written in pure Python, which I did not test. For a program that mostly waits on a network, neither language is the limit.
Concurrency or parallelism: which did I use?
Both programs used parallelism for a CPU-bound job: two processes in Python, two goroutines on two cores in Go. Concurrency alone would not have sped it up. It would have helped a job that waits on I/O.
How do you deploy each to a fleet, and what goes wrong?
Go: build a binary per architecture and copy it. What goes wrong is rollout, signing and versioning. Python: an interpreter at the right version plus packages, usually in a container image. What goes wrong is version mismatch and dependency drift, such as the three different Python versions I had across the lab, the VMs and my build machine.
Which would you use to write a node agent, and which for a benchmark harness?
A node agent runs on every machine, starts often, and must not need anything installed: Go. A benchmark or bring-up harness changes daily and calls ML or measurement libraries: Python. The lab followed the same split.
How do you find a memory problem in each?
Profile it. In Go, use the standard profiler, pprof. In Python, use a profiler such as the standard library’s. I did not hit a memory problem here; both versions held about 10 MB.
Next
Take the quiz to test your understanding: Go vs Python quiz
Part 5 starts a platform layer on the same cluster, beginning with job queues and quotas using Kueue.
Comments