Goal
The earlier posts in Part 5 were about running a shared cluster: queues, scaling, delivery, policy and traffic. This one is about the people who use it. An application developer should not need to write a Deployment, a Service and the right resource limits to run a web page. A platform team can offer a smaller API that says “I want a WebApp with this image”, and build the rest behind it. Crossplane is a way to define such APIs on Kubernetes.
I wanted to define a WebApp resource, build it from a Deployment and a Service, and see how it behaves: how fast it creates things, what the schema refuses, and what happens when someone changes the result by hand. This is the last post in Part 5, the platform layer. The lab has three small nodes and one API, so this shows the mechanism and not a platform.
Terms used in this post
These build on the earlier glossaries.
- Control plane: a program that watches objects and acts to make the world match them. Kubernetes is one, and Crossplane builds more on top of it.
- Composite resource (XR): an object of an API the platform team defined, here
WebApp. - CompositeResourceDefinition (XRD): the definition of that API, including its fields and limits.
- Composition: the recipe that says which objects to create for a composite resource.
- Function: a program that runs inside a composition to produce those objects. I used one that applies patches.
- Provider: a plugin that lets Crossplane create things in some system. Mine creates objects in the same cluster.
- Managed resource: the object a provider reconciles, which in turn creates the real thing.
- Self-service: letting teams create what they need through a supported API, without a ticket.
What I built
- Crossplane 2.4.3. I rendered its Helm chart to manifests and applied them, because the lab has no Helm in the cluster. It was ready in 12 seconds. The core and its helpers used about 235 MiB of memory.
- Two packages. The Kubernetes provider 1.4.1, and the patch-and-transform function 0.11.2. Both were installed and running within about 10 seconds of applying them.
- A narrow identity. The provider runs as a service account, and I bound it to a role that can only manage Deployments and Services. A broader grant would have been simpler and would let anything that can write a WebApp create almost any object.
- The API. A namespaced
WebAppwith a requiredimageand areplicasfield that defaults to 2 and may be 1 to 5. - The composition. It turns a WebApp into a Deployment with a CPU request and a memory limit, and a Service, in the namespace of the WebApp.
Diagram: the developer’s object, the composition, the provider’s managed objects, and what lands in the cluster.
How it works
When a WebApp is created, Crossplane finds the composition for that kind and runs its function, which returns two managed objects with the WebApp’s name, namespace, image and replica count patched in. The provider reconciles each one, creating the Deployment and the Service. The platform team’s choices, such as the memory limit, are in the composition, so every WebApp gets them without the developer writing them.
The developer’s file is five lines:
apiVersion: lab.example.org/v1alpha1
kind: WebApp
metadata: {name: shop, namespace: team-a}
spec: {image: <image>, replicas: 3}
Commands and results
- Create. The Service existed 2 seconds after I applied the WebApp, three pods were ready by the next check, and the WebApp reported Ready about a second later. The image was already on the nodes.
- Change. Raising
replicasfrom 3 to 5 had five pods ready within about a second of my check. - Schema.
replicas: 9was refused with “should be less than or equal to 5”, and a WebApp with no image was refused with “Required value”. - Delete. Deleting the WebApp removed the Deployment and the Service within seconds.
- Changing the result by hand. I deleted the Deployment that Crossplane had created. In five trials the Deployment was back with all its pods in about 30 seconds twice (31 s and 28 s), and not back within 240 to 300 seconds three times. The provider-kubernetes v1.4.1 source sets the drift poll interval,
--poll, to 10 minutes by default with 10 percent jitter, and this provider ran with its default arguments. A deletion that nothing else triggers is therefore noticed at the next poll, which can be later than the four to five minutes I waited, so the three slow trials fit the setting. The two quick recreations are the ones the poll interval does not account for.
Screenshot: the output of the drills, ending with the five trials of putting a deleted Deployment back. The first drill’s wait for the Deployment ended without it, which is the first of the three trials that did not come back.
Challenges
- Putting things back is not reliable here. Argo CD in the previous post undid a manual change within about a second. Crossplane’s provider put the Deployment back in about 30 seconds in two trials and not within four to five minutes in three. A platform that promises to keep resources as declared needs a known and short correction time, and the setting that controls this one is the provider’s
--pollflag, which defaults to 10 minutes. - Indirection makes failures harder to trace. A problem could be in the WebApp, the composition, the function, one of two managed objects, the Deployment or its pods. I used
kubectl geton the managed objects to see what had been created. In a fleet, the debugging path has to be documented and the status has to be surfaced on the WebApp. - The identity is part of the product. The provider needed permission to create Deployments and Services, and I had to find its service account, whose name has a random suffix, to bind the role to it. A provider that can create anything is a way around the cluster’s access rules for anyone who can write a WebApp.
- Versions changed the model. Crossplane 2 makes composite resources namespaced and uses a different API group for the provider’s objects than many examples show. Examples written for version 1 would not have worked, so the version of the documentation matters.
What this lab cannot show is many teams using many APIs, upgrades of the packages, or what happens when a composition changes under live WebApps.
Differences with a production setup
- Cloud resources. The same pattern is used for databases, buckets and queues through providers for AWS, Google Cloud and Azure, listed on the Upbound marketplace. My provider only creates objects in the same cluster.
- Compositions. Production compositions are usually written as functions in a programming language, with tests, and not as YAML patches.
- Delivery. WebApps would be applied from git by a tool such as Argo CD, so that requests are reviewed and recorded.
- Portals. A developer portal such as Backstage can put a form in front of these APIs.
- Policy. A policy engine like the one in an earlier post would constrain what a WebApp may ask for, beyond the schema limits.
- Terraform. Terraform from Part 4 does a similar job from outside the cluster, with a state file and a plan. Crossplane runs inside the cluster and reconciles continuously.
FAQ
What does Crossplane do that Helm or Argo CD does not?
Helm templates objects and Argo CD applies them from git. Crossplane lets the platform team define a new API on the cluster and keeps the objects behind it reconciled. They are often used together.
What is a composition?
The recipe that turns one composite resource into the objects behind it. Here it makes a Deployment and a Service from a WebApp.
How is self-service different from a ticket queue?
The developer creates the resource through the supported API and the platform’s rules, such as limits, apply automatically. No one has to do it for them.
What limits can the platform put on the API?
The schema: required fields, defaults and ranges. Mine refused 9 replicas and a missing image. Anything else goes into the composition or a policy.
What did the manual-change test show?
That correction was not predictable in this setup: about 30 seconds in two trials, and not within four to five minutes in three. The poll interval and whatever else triggers a reconcile would need to be understood before relying on it.
Next
Take the quiz to test your understanding: Part 5 quiz
Part 5 ends here. Part 6 moves to an emulated bare-metal fleet.
Comments