Goal
In Part 2 I reached the web page through a Service. A real platform needs more at the edge: routing by host name or header, releasing a new version to a fraction of users, and limiting which workloads may talk to which. The Kubernetes Gateway API is the current way to describe the first two, and a network policy is the way to do the third.
I wanted to turn on the Gateway API in the ingress controller that k3s already ships, route traffic between two versions of a page, and then lock the pods down so that only the gateway can reach them. This is the fifth post in Part 5, the platform layer. The lab has three small nodes, so this shows how the pieces behave and says nothing about scale.
Terms used in this post
These build on the earlier glossaries.
- Ingress: an older Kubernetes object for routing HTTP traffic into a cluster. The Gateway API replaces it with a richer model.
- GatewayClass, Gateway, HTTPRoute: the Gateway API objects. The class names a controller, the Gateway opens a listener, and a route says where requests go.
- Weighted routing: sending a share of requests to each backend, such as 90% and 10%.
- Canary release: sending a small share of traffic or selected users to a new version first.
- Network policy: a rule that says which pods may connect to which. Without one, every pod can reach every other pod.
- CNI (container network interface): the plugin that provides pod networking and, in some cases, enforces network policy.
What I built
- The controller. k3s ships Traefik 3.7.15 and the Gateway API definitions, but the Gateway provider was off. I turned it on by adding a small
HelmChartConfigfor the bundled chart. About 3 seconds after that, a GatewayClass namedtraefikexisted, and Traefik restarted to pick up the change. It was accepted about 15 seconds after applying. - Two versions of a page. Two nginx Deployments of two pods each,
v1andv2, in one namespace, each with a Service and each serving a page that names its version. - A Gateway and a route. One listener on HTTP. One route for the host
hello.lab.test: requests with the headerx-canary: yesgo tov2, and all others are split 90% tov1and 10% tov2. - Network policy. A rule that denies all incoming connections to the pods in that namespace, and a second rule that allows only the Traefik pods to reach port 80.
Diagram: the client, the Gateway and its route, the two versions, and a client in another namespace that the policy refuses.
How it works
The Gateway API objects only describe what should happen. Traefik watches them and configures itself, and it is the component that receives the traffic on each node. The route’s rules are checked in order: the header rule first, then the weighted rule. The network policy is enforced by the network plugin on each node. k3s includes a policy controller, and once a pod is selected by any policy, only what the policies allow reaches it.
Commands and results
From my own machine I sent requests to a node’s address with a Host header.
| Request | Result |
|---|---|
| 300 requests, default route | 270 from v1, 30 from v2 |
50 requests with x-canary: yes | all 50 from v2 |
| 5 requests with another host name | 5 times 404 |
| 100 requests after changing the weights to 50/50 | 50 from v1, 50 from v2 |
300 requests with v2 scaled to zero pods and the weights at 50/50 | 150 from v1 and 150 503 errors |
Screenshot: a second round of the tests with a dashboard open, at about 12 requests per second. First the 90/10 split, then the canary header sending everything to v2, then 50/50 with v2 down, when v1 gets about 8 requests per second and the entry point returns 503 for the rest, then the restored split. The share of v2 drops from 100% to zero in that third phase.
Network policy. A test pod in the default namespace fetched the v1 Service. Before the policies it got the page. With the policies applied it got “Connection refused”, and the request through the Gateway still returned the page. After I removed the policies, the test pod got the page again.
Challenges
- A weighted route does not fail over. With
v2scaled to zero, the route still sent half the requests to it, and half the requests failed with503. A canary that loses all its pods turns directly into errors for its share of users. In a fleet, a release needs a health check on the new version and a way to remove it from the route automatically. - The shared edge is a shared risk. Turning on the Gateway provider changed the configuration of Traefik, which is also the ingress for everything else in the cluster, and it restarted the pod. A change to the edge has a blast radius larger than the application it was made for, so it needs more than one replica and a staged rollout. The lab runs one Traefik pod, and I did not measure whether requests failed while it was replaced.
- Default deny changes what else can connect. The policy blocked every other pod, including tools I might forget. I did not test what it does to monitoring that scrapes those pods, which in a real namespace has to be allowed explicitly.
- A policy depends on the network plugin. It worked here because k3s’s networking enforces it. A network plugin that does not enforce policies would accept the objects and ignore them, so the first test of any policy should be a connection that has to fail.
What this lab cannot show is encrypted traffic with real certificates, many routes owned by different teams, or an edge that has to survive the loss of a node.
Differences with a production setup
- Ingress versus Gateway. The Ingress object is still common. The Gateway API separates the roles of the platform team, which owns the Gateway, and application teams, which own their routes. I put both in one namespace.
- Controllers. Traefik is one implementation, described in its Gateway documentation. Others exist, and the same objects work with them.
- Certificates. Production edges terminate TLS with certificates issued and renewed by a tool such as cert-manager. I used plain HTTP.
- Policy engines. Cilium enforces network policy by workload identity and can filter at layer 7. I used the basic Kubernetes objects.
- Service meshes. Istio adds encrypted pod-to-pod traffic and fine-grained traffic control. I used neither.
- Progressive delivery. Splitting traffic by hand is the manual form of what Argo Rollouts automates, with analysis and automatic rollback.
- Load balancing. A cloud load balancer sits in front of the nodes. Mine is k3s’s built-in service load balancer on the node addresses.
FAQ
What does the Gateway API do that Ingress does not?
It expresses routing by header, weight and other criteria in the standard objects, and splits ownership between the team that runs the Gateway and the teams that own routes.
What is a canary release?
Sending a small share of traffic, or selected users, to a new version before everyone. Here, 10% by weight, or any request with a canary header.
Why did half the requests fail when v2 had no pods?
The route still sent that share to v2, which had no ready endpoints, and the gateway returned 503. It does not move the share to v1 on its own.
What is the default network policy behaviour?
With no policy, any pod can reach any other pod. Once a policy selects a pod, only the traffic that a policy allows can reach it.
How do I know a network policy is really enforced?
Test a connection that should fail. My client in another namespace was refused, and it worked again after I removed the policies.
Next
Take the quiz to test your understanding: Part 5 quiz
The last post in Part 5 shows how a platform team can offer developers a simple resource that creates other resources for them, using Crossplane.
Comments