A Go CLI tool for the cluster: capacity and model server status

Goal

In Part 2 I worked out requested and allocatable resources by hand with kubectl. A fleet needs that as something a script or another person can run, so I wrote it as a small Go tool, labctl, using the same Kubernetes API that kubectl uses. It reports each node’s requested resources against what the node has, and it reports the model server’s status.

This is the last post in Part 4. It follows the foundations post on Go and Python, which explains why a tool like this is usually written in Go.

Terms used in this post

These build on the earlier glossaries.

  • API server: the Kubernetes component that every tool talks to. It answers requests to read or change objects.
  • client-go: the official Go library for talking to the Kubernetes API.
  • Allocatable: the CPU and memory on a node that pods can use, after what the node reserves.
  • Request: the amount of CPU or memory a pod asks the scheduler to set aside. The scheduler places pods by requests, not by use.
  • ServiceAccount: an identity for a program inside the cluster.
  • RBAC (role-based access control): the rules that say which identity may do which action on which kind of object.
  • Unit test: a test of one piece of code that does not need the rest of the system.

What I built

labctl has two commands:

  • labctl capacity prints, for every node, how many pods are placed on it and the sum of their CPU and memory requests against the node’s allocatable amounts.
  • labctl status prints the model server’s desired, ready and available replicas, how many pods run on each node, and which pods are not ready. Both commands can print JSON.

The code is split in two. main.go parses flags, builds the client and prints. A report package does the calculations on plain Kubernetes objects, so it can be tested without a cluster. It skips pods that have finished, because the scheduler does too. It does not count init containers or pod overhead, which the README says.

It runs with its own identity. I created a ServiceAccount with a role that can only get and list nodes, pods and Deployments, and gave the tool a token that lasts 24 hours.

How labctl reads the cluster Diagram: labctl sends read-only requests to the API server, and the report package turns the objects into the two tables.

How it works

capacity lists all nodes and all pods, then adds up the requests of the pods on each node. status reads the Deployment, uses its label selector to list its pods, and counts them by node and by whether their Ready condition is true. The numbers come straight from the API, so the tool adds no state of its own.

Commands and results

With the Ray cluster, four model server replicas and the monitoring stack running, and no load, labctl capacity printed:

NodePodsCPU requestedMemory requested
lab-node171620m of 2000m (81%)1832 Mi of 4929 Mi (37%)
lab-node281720m of 2000m (86%)1728 Mi of 4929 Mi (35%)
slurm-ctl101820m of 2000m (91%)2124 Mi of 4929 Mi (43%)

kubectl describe nodes gave the same request figures for all three nodes. labctl status printed model-server: 4/4 ready, 4 available, with two pods on lab-node1 and one each on lab-node2 and slurm-ctl.

The three unit tests cover the sums and the skipping of finished pods, a node with no allocatable CPU, and the count of ready pods per node. They run in a few milliseconds and need no cluster.

I also checked the identity. Asked as the ServiceAccount, the cluster said it cannot delete pods and cannot list Secrets.

The binary is 27 MB with symbols stripped. The Go log parser in the previous post was 1.9 MB, so most of this size comes from the Kubernetes client library.

Challenges

  • Requested is not used. The nodes show 81% to 91% of their CPU requested while nothing was running. Each Ray worker asks for 1000m, and together with the model server and the monitoring stack the requests leave little room, even though the real use was low. In a fleet this is why a cluster can look full to the scheduler and idle on its dashboards. The tool shows only requests, so it cannot show the gap, and I did not add usage to it.
  • A scoped identity has an expiry. A read-only token that lasts 24 hours is safer than the administrator credentials, and it also stops working after a day. Whatever runs a tool like this on a schedule needs a way to renew its credentials.
  • A report is only as complete as its rules. The tool ignores init containers and pod overhead, so on a cluster that uses them its numbers could differ from the scheduler’s. They matched here because the lab does not use them.

What this lab cannot show is a large cluster, where listing every pod on each call is slow, or a cluster with several teams and namespaces.

Differences with a production setup

  • Reading the API. A tool that runs all the time uses informers, which watch for changes and keep a local cache, instead of listing every pod on each call. labctl lists everything once per run.
  • Actual use. metrics-server or Prometheus supplies what pods are really using, which is needed to compare with the requests.
  • Packaging. A tool like this is often shipped as a kubectl plugin through Krew, or built into an operator with the Kubebuilder framework.
  • Testing. Controllers are usually also tested against a real API server with envtest. I tested only plain functions.
  • Identity. A production tool gets a short-lived credential from the platform’s identity provider, under RBAC rules reviewed like code, and does not use a token I made by hand.
  • Reserved resources. Real nodes reserve compute for system daemons, so allocatable is lower than capacity. In this lab, k3s reserves nothing by default.
  • Right-sizing. Tools use real usage to adjust requests, such as the Vertical Pod Autoscaler, the descheduler that moves pods, and Karpenter, which can consolidate under-used nodes. labctl only reports.

FAQ

Why write this in Go and not a shell script around kubectl?

A script would work for one cluster. A Go program has types for the Kubernetes objects, a test suite, and one binary to hand to someone. The cost is the size: 27 MB here.

What is the difference between a request and a limit?

The request is what the scheduler sets aside when it places the pod. The limit is the most the pod may use. This tool sums requests.

Why does the tool skip finished pods?

The scheduler stops counting them, and a report that counted them would show the node fuller than the scheduler believes it is.

Why a separate ServiceAccount?

So the tool can do exactly what it needs and nothing else. It cannot delete anything or read Secrets, and I tested both.

Next

Take the quiz to test your understanding: Part 4 quiz

Part 5 starts a platform layer on the same cluster, beginning with job queues and quotas using Kueue.

Comments