Kubernetes 2.5: a Go admission webhook and controller

Goal

Earlier posts used admission policy from tools that other people wrote: Kyverno in the supply-chain post and the built-in pod security levels in the previous one. This post writes the pieces myself, in Go, to see how they work and how they fail. I built a validating admission webhook and a small controller, ran both on the cluster, and then broke them on purpose: the webhook with no running pod, with a slow answer and with a rule that covers its own namespace, and the controller with a deleted object and a frozen leader.

Terms used in this post

  • Admission webhook: a web service that the API server calls when an object is created or changed. It answers allowed or denied.
  • ValidatingWebhookConfiguration: the object that tells the API server which requests to send to which webhook.
  • failurePolicy: what the API server does when the webhook cannot be reached or does not answer in time. Fail refuses the request and Ignore lets it through.
  • Controller: a loop that compares what exists with what should exist and fixes the difference.
  • Reconcile: one pass of that loop for one object.
  • Informer: a client-go component that keeps a local copy of objects and tells the controller when they change.
  • Lease: a small object that holds a name and a renewal time. Controllers use it for leader election.
  • Leader election: with several replicas of a controller, only the one that holds the lease does the work.

A request through admission Diagram: where the webhook sits in a request, and what each kind of answer or non-answer does.

What I built

One Go program, kp6, with two modes. It uses client-go and the Kubernetes API types and nothing else. I wrote it in Go 1.27 on the arm64 server, built a static amd64 binary, put it in an image with nothing else in it, and loaded the image into containerd on each node. That is not how a production cluster gets images.

The webhook serves HTTPS and handles one kind of request. A pod is refused unless it has an owner label. The decision is one small function:

func review(req *admissionv1.AdmissionRequest) *admissionv1.AdmissionResponse {
	resp := &admissionv1.AdmissionResponse{UID: req.UID, Allowed: true}
	var pod corev1.Pod
	if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
		resp.Allowed = false
		resp.Result = &metav1.Status{Code: http.StatusBadRequest, Message: "cannot read the pod: " + err.Error()}
		return resp
	}
	if pod.Labels["owner"] == "" {
		resp.Allowed = false
		resp.Result = &metav1.Status{Code: http.StatusForbidden, Reason: metav1.StatusReasonForbidden, Message: ownerMessage}
	}
	return resp
}

A table test covers five label combinations and a pod that cannot be read, and it passes. The webhook also reads its certificate from the mounted file again whenever the file changes. A second path, /slow-validate-pods, waits 15 seconds before it answers, so that I can test a webhook that is too slow.

The controller keeps one ConfigMap, owner-defaults, in every namespace labelled kp6-guard=on. It watches namespaces and ConfigMaps through informers, puts the namespace name on a work queue, and in each pass creates the ConfigMap if it is missing or puts the data back if it differs. Two replicas run. A Lease named kp6-controller decides which one works. The lease lasts 15 seconds, the leader must renew within 10, and a standby retries every 2.

The setup has the certificate for the webhook’s service issued by cert-manager from the private CA of the previous post. The same tool copies the CA into the webhook registration, so I did not paste any certificate by hand. The registration covers pod creation in namespaces labelled kp6-guard=on only, with a five second timeout and failurePolicy: Fail. One namespace, guarded, carries the label.

How it works

When a request to create a pod reaches the API server, it has been authenticated and authorized. Admission then calls each matching webhook with an AdmissionReview, which holds the object. The webhook returns the same review with a response in it. The API server stores the object only if every webhook allows it.

A webhook is part of the path of every matching request. If it is slow, the request is slow. If it does not answer, the API server falls back to failurePolicy. That setting is a decision about which is worse for the cluster: refusing requests while the check is broken, or letting through requests that were never checked.

A controller works the other way. It does not sit in the path of a request. It watches, and it acts after the fact. Running two replicas keeps it available, and the lease makes sure that only one of them acts at a time.

Commands and results

The rule. A pod without an owner label in guarded was refused with the webhook’s message. A pod with owner=alice was created. A pod without a label in default, which is not guarded, was created.

Webhook down, Fail. I scaled the webhook to zero. Creating a pod in guarded failed with failed calling webhook "owner-label.kp6.lab" and no endpoints available for service "kp6-webhook", even for a pod with an owner label. A pod in default was not affected, because the registration only covers the labelled namespaces.

Webhook down, Ignore. I changed the policy to Ignore. A pod without an owner label was created in guarded. The rule was off, and nothing said so.

Webhook too slow. I brought the webhook back and pointed the registration at the slow path. With Fail, a pod with an owner label was refused after 5.2 seconds with context deadline exceeded. With Ignore, a pod without a label was created after the same 5.2 seconds. The timeout was five seconds. In both cases the request was held for that long.

Admission results Image: the rule, a webhook with no pod, and a slow webhook, under each failure policy.

A webhook in its own way. I added a second registration with Fail that also covered kp6-system, the namespace where the webhook runs, and deleted the webhook pod. The ReplicaSet could not create the replacement, because creating it needed the webhook to approve it, and the webhook was the missing pod. After 20 seconds there were no webhook pods. Deleting the extra registration was the way out, and the pod was running about four seconds later.

The webhook that blocked its own restart Image: the replica set’s event while the webhook could not start.

The controller. Deleting the ConfigMap in guarded brought it back in under a second, and so did changing its data. With the leader pod deleted, the work resumed 5.1 seconds later, because the old leader released the lease when it was told to stop. For a leader that hangs, I froze the leader’s process with SIGSTOP so that it could not renew. I deleted the ConfigMap three seconds after the freeze and it came back 19.5 seconds after the freeze, once the lease had run out and another pod held it.

Controller results Image: reconcile times and the two kinds of leader loss.

The webhook’s certificate. The certificate for the webhook has a one-hour duration, so that cert-manager renews it every five minutes. The same webhook pod served the first certificate and then, after a renewal, a second certificate with a different serial number and file time. Nothing restarted. This is the case from the last post where a server kept its old certificate until a reload. Here the program reads the file again whenever it changes, and so it needs none.

The webhook's certificate before and after a renewal Image: two log lines from the same pod.

Challenges

  • A failing webhook stops the cluster in the namespaces it covers. With Fail, a webhook with no pods refused every pod in guarded, including correct ones. With Ignore, the rule vanished without a sign. A webhook that matters is run with several replicas, spread over nodes, with a disruption budget, and alerts on the failure of its calls.
  • A webhook has to stay out of its own way. The registration that covered the webhook’s namespace made the webhook impossible to restart. Production registrations exclude the namespaces of the webhook and of the control plane, and have a documented way to remove them when something goes wrong.
  • A slow webhook is a slow cluster. Every matching request waited the full timeout, 5.2 seconds. The default timeout is longer, and a webhook on a busy path can add that delay to every create.
  • Failover takes as long as the lease. A clean stop handed the work over in 5 seconds. A hung leader took 19.5 seconds, which is the lease duration plus the retry. Work done by the controller stops for that long. A shorter lease is faster and more likely to switch leader on a short pause.

What this lab cannot show is how a webhook behaves under real load, or how an API server with many webhooks adds up their delays. I sent a few requests at a time. In the future I’m planning to purchase a hardware kit to help practice further.

Differences with a production setup

  • Frameworks. I used client-go directly to see each step. Most operators use controller-runtime and Kubebuilder, which provide the queue, the caching, the leader election and the webhook server.
  • Custom resources. My controller manages a ConfigMap that I chose. Operators usually define their own resource type and reconcile that. The Kubernetes page on controllers describes the pattern.
  • Leader election. I used the leaderelection package with a Lease, as production controllers do.
  • Rules without a webhook. ValidatingAdmissionPolicy runs rules written in CEL inside the API server, with no web service to keep alive. For my owner label rule that would have been enough. A webhook is for checks that need code or outside data.
  • Policy engines. Kyverno and Gatekeeper are webhooks that other people maintain and test. They have the same failure modes that I measured.
  • Webhook registration. The Kubernetes page on dynamic admission control lists the settings I used and others, such as match conditions and object selectors, which narrow what is sent.
  • Certificates. I used cert-manager and its CA injector to keep the webhook’s certificate and the CA bundle in step.
  • Images. I loaded the image by hand into each node. Production setups push it to a registry, and sign it, as in the supply-chain post.

FAQ

Should a webhook use Fail or Ignore?

It depends on what the check protects. A check that prevents damage is usually Fail, with a webhook that is built to stay up. A convenience check is usually Ignore.

Why not put every rule in a webhook?

Each one adds a service that has to be up and fast. ValidatingAdmissionPolicy covers many rules with no service at all.

Does a controller need leader election?

A single replica does not. Two or more replicas that act at once can undo each other’s work, so replicas elect one to act.

Why did the frozen leader take longer to replace than a deleted one?

A deleted leader gives the lease back when it stops. A frozen one cannot, so the standby has to wait for the lease to expire.

Repeating with real hardware

I would run the webhook with three replicas on three nodes and a disruption budget, drain a node during a burst of pod creation, and time the effect on requests. I would also run the controller on a cluster with thousands of namespaces to see how long the first reconcile takes after a leader change.

Next

Take the quiz to test your understanding: Kubernetes quiz

The last planned post in this track covers cluster lifecycle: building a cluster from nothing with kubeadm and the tools that automate it.

Comments