Goal
A cluster that anyone with access can deploy to will, sooner or later, run something it should not: an image with no tag, a container with no memory limit, an image nobody built. Admission policy puts checks in the path of every create request, before the object exists. Supply-chain checks go further and ask where an image came from.
I wanted to run Kyverno on the lab cluster, report on what already runs, enforce rules in one namespace, and then require that images from the lab’s registry are signed and come with a software bill of materials. This is the fourth post in Part 5, the platform layer. The lab has three small nodes and one registry, so this shows the mechanism and not a full supply chain.
Terms used in this post
These build on the earlier glossaries.
- Admission control: checks that run when an object is created or changed, before it is stored. They can allow, deny or modify it.
- Policy as code: rules written as objects that are kept in git and applied like any other manifest.
- Audit and Enforce: Audit reports violations and allows them. Enforce denies them.
- Container registry: a server that stores images.
- Digest: a hash that identifies exactly one image. A tag such as
1.0can be moved to another image, and a digest cannot. - Image signing: attaching a cryptographic signature to an image so that a verifier can tell who published it and that it has not changed.
- SBOM (software bill of materials): a list of the packages inside an image, in a standard format such as SPDX.
- Attestation: a signed statement about an image, such as its SBOM.
What I built
- Kyverno 1.19.2, installed from its release manifest in 26 seconds. It runs four controllers and used about 230 MiB of memory.
- Two policies for pods outside the system namespaces: no
latestor missing image tag, and a memory limit on every container. They report everywhere and deny in one test namespace. - A registry. The lab had none, so I ran a small registry in the cluster over plain HTTP, reachable on a port of every node. Each node needed a small configuration file so its container runtime would pull from it, and k3s restarted to read it.
- A signed image. I pushed an nginx image to the registry, signed it with cosign using a key pair I made, made an SBOM with Syft, and attached the SBOM as a signed attestation.
- A verification policy. For images from the lab registry, Kyverno requires a valid signature and an SBOM attestation, and pins the image to its digest.
Diagram: signing and attaching an SBOM on the build side, and the checks Kyverno makes when a pod is created.
How it works
When a pod is created, the API server sends it to Kyverno’s webhook before storing it. Kyverno compares it with each policy. For image verification it contacts the registry, finds the signature and the attestation for the image’s digest, checks them against the public key in the policy, and rewrites the image reference to include the digest. A pod that fails an Enforce rule is rejected with a message that names the policy and the rule. In Audit mode it is created and the failure is recorded in a policy report.
Commands and results
Audit first. After the two policies were applied in Audit mode, the background scan covered 14 pods and reported no violations, because the workloads I had written already set limits and tags. A test pod with no memory limit in the default namespace was created, and the policy report then showed require-limits: fail for it.
Enforce in a test namespace.
| Pod | Result |
|---|---|
| No memory limit | Denied: “Every container must set a memory limit” |
Image tagged latest | Denied: “Images must have an explicit tag that is not ‘latest‘“ |
| Both fixed | Created |
Screenshot: the cluster’s answers for the four test pods. The two denials name the policy and the rule, and the last pod, in an audit-only namespace, is created and then reported as a failure.
Signed images. Each pod used an image from the lab registry unless noted.
| Image | Result |
|---|---|
| Signed, with an SBOM attestation | Created and running. The stored pod spec had the digest added to the image reference |
| Not signed | Denied: “no signatures found” |
| Signed, no SBOM attestation | Denied: “no matching attestations” |
| Not signed, in another namespace | Denied, because this policy applies everywhere |
| The signed tag overwritten with a different image | Denied: “no signatures found”, because the signature belongs to the old digest |
| An image from outside the lab registry | Created, because the policy does not cover it |
Screenshot: the first pod is admitted and its image reference carries the digest. The unsigned image and the image without an SBOM attestation are denied, as is the unsigned image in another namespace. The image from outside the lab registry is created.
The SBOM listed 69 packages in SPDX 2.3 format.
Challenges
- Audit found nothing, and that proves little. A clean report on 14 pods says my own workloads follow two simple rules. In a real fleet the first audit usually lists many violations, and the work is deciding which to fix and which to exempt before enforcing.
- Digest pinning needs Enforce. Kyverno refused the verification policy in Audit mode because it also rewrites the image to its digest, and that rewrite is a change. I had to enforce it, and limited it to images from the lab registry so it could not block anything else.
- The registry needed node changes. Pulling from a plain HTTP registry meant a configuration file on all three nodes and a restart of k3s on each. After the server restart the API answered again within 12 seconds and the nodes reported Ready. The restart is a cluster maintenance task that a registry change forces, and I did not check every workload during it.
- Tags are not identity. Overwriting the signed tag with another image made the new content fail verification, which is the point of signing digests. It also means a deploy that refers to a tag can break when the tag moves.
- A blocked create is quiet for controllers. A Deployment whose pods are denied shows no ready pods and records events on its ReplicaSet. I tested bare pods and did not test a Deployment.
- Keys are a process. I made one key pair without a passphrase and kept the private key outside the repository. In a fleet, who holds the signing key, how it rotates and what happens when it leaks matter more than the commands.
What this lab cannot show is policy for hundreds of teams, an outage of the policy engine, or scanning for known vulnerabilities, which an SBOM makes possible and which I did not do.
Differences with a production setup
- Policy engines. Kyverno is one; Gatekeeper is another, and Kubernetes has a built-in ValidatingAdmissionPolicy. Kyverno itself prints that the policy kind I used is deprecated in favour of newer policy types.
- Keyless signing. Production setups often sign with short-lived identities and record signatures in a public transparency log through Sigstore. I used a key pair and turned the transparency log off.
- Provenance. SLSA describes how a build should prove how an image was made, and an SBOM is only one piece of that.
- Registries. A real registry has accounts, access control and replication, as in Harbor. Mine is the open source distribution registry with storage that disappears on restart.
- Vulnerability scanning. An SBOM is used with a scanner such as Trivy to find known flaws. I did not run one.
- Availability. A policy webhook that fails closed blocks pod creation when it is down. Production runs several replicas and tests that case, which I did not.
FAQ
What is an admission controller?
A check that runs on a create or update request after the API server authenticates it and before it stores the object. It can allow, deny or change the request.
What is the difference between Audit and Enforce?
Audit lets the request through and records the violation. Enforce rejects it. Teams start with Audit to find out what would break.
Why sign the digest and not the tag?
A tag is a movable label, and a digest identifies content. Overwriting the signed tag with another image failed verification, because the new image’s digest had no signature.
What is an SBOM for?
It lists what is inside an image, so you can answer which images contain a package when a flaw is announced. Mine listed 69 packages for an nginx image.
Does a signature mean an image is safe?
No. It says who published the image and that it has not changed. The image can still contain flaws, which is why scanning and provenance are separate steps.
Next
Take the quiz to test your understanding: Part 5 quiz
The next post in Part 5 puts traffic through the Gateway API and restricts which pods can talk to each other with network policy.
Comments