Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 78 ยท Security

Supply Chain and Cluster Hardening

Secure what you run and where it runs: image signing and scanning, imagePullSecrets, etcd encryption, audit logs and the CIS benchmark, with a hardening checklist in Python.

So far this chapter has been about who may talk to the cluster (RBAC), what pods may do (security contexts) and what content is accepted (admission policy). Two big areas remain. The first is the software supply chain: where do the images you run come from, and can you prove they are the ones you built? The second is cluster hardening: making the control plane, the nodes and the data stored in etcd harder to attack and easier to investigate afterwards.

Nothing here runs on this site, so commands and manifests are read-only. The interactive parts are small Python audits that show how automated checkers (a signature gate, a configuration scanner) reason.

The supply chain in one picture

From source to running pod

Build CI builds the image from reviewed code.
Scan Look for known vulnerabilities.
Sign Attach a signature and an SBOM.
Verify The cluster checks before running.

Every step can fail, and an attacker only needs one weak link.

1 / 2

Images: pin, scan, sign

Pin by digest. A tag like web:1.4 is a movable label; whoever controls the registry can point it at different content tomorrow. A digest (web@sha256:9f86...) names exact content. Many teams keep a readable tag for humans and deploy by digest:

YAML
containers:
  - name: web
    image: registry.example.com/shop/web:1.4@sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

When both a tag and a digest are given, the digest wins. Tools like Renovate and Argo CD Image Updater can bump pinned digests automatically.

Scan. Vulnerability scanners such as Trivy and Grype compare the packages inside an image with databases of known flaws. Run them in CI on every build and on a schedule for images already in your registry, because new flaws are published for old software every day. Keep images small (distroless or slim bases): fewer packages, fewer findings. See the Docker CI pipelines lesson for where scanning sits in a pipeline.

Produce an SBOM. A software bill of materials lists every component in the image. It lets you answer "are we affected?" within minutes when a new library flaw makes the news.

Sign. A signature proves that a specific identity (your CI pipeline) produced a specific digest. With the Sigstore tool cosign, a pipeline can sign "keylessly" using its OIDC identity, and verification checks which identity signed it:

CONSOLE
$ cosign sign registry.example.com/shop/web@sha256:9f86...
$ cosign verify registry.example.com/shop/web@sha256:9f86... \
    --certificate-identity "https://github.com/example/shop/.github/workflows/release.yml@refs/heads/main" \
    --certificate-oidc-issuer "https://token.actions.githubusercontent.com"

Kubernetes does not verify signatures on its own. You add the check through an admission policy engine: Kyverno has a verifyImages rule, and the Sigstore policy-controller and Ratify are other options, as covered in Admission Control and Policy. With such a rule, an unsigned or wrongly-signed image never starts.

Private registries and imagePullSecrets

Your private images need credentials. Kubernetes stores them in a Secret of type kubernetes.io/dockerconfigjson:

CONSOLE
$ kubectl create secret docker-registry regcred \
    --docker-server=registry.example.com \
    --docker-username=ci-reader \
    --docker-password="$REGISTRY_TOKEN" \
    -n shop

A pod uses it with imagePullSecrets. Rather than repeat that in every pod, attach it to the ServiceAccount, and every pod that uses the account inherits it:

YAML
apiVersion: v1
kind: ServiceAccount
metadata:
  name: web
  namespace: shop
imagePullSecrets:
  - name: regcred

Secrets are namespaced, so each namespace needs its own copy. Use a read-only, pull-only registry token (never a developer's password), rotate it, and prefer cloud-native nodes that authenticate to the registry through their cloud identity (for example an EKS node role that can read ECR), so no pull secret is needed. For registry basics, see the Docker lessons on private registries. A pull failure shows up as ErrImagePull or ImagePullBackOff in kubectl get pods.

Encrypting Secrets in etcd

All cluster state lives in etcd, including every Secret. By default, Secrets are stored there base64-encoded, which is not encryption: anyone who can read the etcd data or a backup can read your passwords (see the upcoming lesson on Secrets). Encryption at rest tells the API server to encrypt chosen resources before writing them:

YAML
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:
          apiVersion: v2
          name: my-kms
          endpoint: unix:///var/run/kms/socket.sock
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded 32-byte key>
      - identity: {}

The first provider is used to write; all listed providers can read, so you can rotate by adding a new key at the top and then re-writing every secret (kubectl get secrets -A -o json | kubectl replace -f -). A KMS provider is the strongest option, because the encryption key lives outside the cluster in a cloud key service or HSM. On managed Kubernetes (EKS, GKE, AKS) you usually switch this on with one setting. After enabling it, verify by reading the raw value with etcdctl: an encrypted value begins with k8s:enc:.

Audit logs

RBAC tells you what can happen; an audit log records what did. The API server writes one event per request stage, and an audit policy decides how much detail to record per rule. The levels are None (skip), Metadata (who, what, when, no bodies), Request (adds the request body) and RequestResponse (adds the response body too).

YAML
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
  - level: Request
    verbs: ["create", "update", "patch", "delete"]

Rules are evaluated top to bottom and the first match wins. Record Secrets at Metadata only, never Request or RequestResponse, or the secret values end up in the log. Ship the log off the node to a place attackers cannot edit, and alert on things like pods/exec in production, new ClusterRoleBindings, or access to Secrets by unexpected users. Managed clouds offer the same data through their logging services.

The stand-in below mimics how the audit policy picks a level for an event with that first-match rule.

Python Audit policy: the first matching rule wins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that reading a secret is logged at Metadata, but because that rule comes before the generic "writes" rule, even a secret write never reaches Request level. That ordering is deliberate: it keeps secret values out of the log.

Hardening the cluster and the CIS benchmark

The CIS Kubernetes Benchmark is a long, freely available list of recommended settings for the API server, etcd, kubelet and workloads. The tool kube-bench (run as a Job) tests a node against it, and managed providers publish how they map to it. Typical items:

  • API server: anonymous auth disabled, --authorization-mode includes Node,RBAC, audit logging enabled, admission plugins such as NodeRestriction turned on.
  • Kubelet: --anonymous-auth=false, webhook authorization instead of AlwaysAllow, --read-only-port=0, rotate certificates.
  • etcd: client certificates required, reachable only from the API servers, encrypted backups.
  • Cluster design: a private API endpoint (or an IP allow list), separate node pools for sensitive workloads, nodes running a minimal OS with automatic patching, and no workloads on control-plane nodes.
  • Workloads: the Restricted Pod Security level where possible, network policies with default deny, and no cluster-admin for apps.

Here is an automated checker in miniature. It takes a configuration as a dictionary and reports which controls fail, ranked by severity. Real scanners read the actual flags of running components; the checklist idea is identical.

Python A hardening checklist runner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A third quick stand-in shows the image gate from earlier: a function that decides whether a pod's image reference is acceptable (pinned by digest, from an allowed registry, signature on the approved list).

Python An image admission gate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is real: digests, cosign signatures, EncryptionConfiguration, audit policies and the CIS benchmark all work exactly as shown on real clusters. What is the stand-in: the Python code, which checks dictionaries instead of live components and pretends signatures were already verified.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is pinning an image by digest safer than using a tag?

    A digest is a hash of the image content. A tag is just a pointer that the registry owner can change.
  2. 2. How do you make every pod of a ServiceAccount pull from a private registry without repeating the secret in each pod?

    Pods inherit imagePullSecrets from their ServiceAccount, so you set it once per account and namespace.
  3. 3. Secrets in etcd are base64-encoded by default. What does that mean for security?

    Base64 is reversible encoding, not encryption. Configure an EncryptionConfiguration (ideally with a KMS provider) to protect Secrets at rest.
  4. 4. In an audit policy, a None rule appears above a Metadata rule that matches the same request. Which level applies?

    Audit rules are checked in order and the first match decides the level.
  5. 5. Which tool tests a node against the CIS Kubernetes Benchmark?

    kube-bench runs the CIS checks and reports pass, fail and warn results.

Summary

  • Secure the supply chain end to end: build in CI, scan, produce an SBOM, sign, and verify at admission. Pin images by digest.
  • Private images need imagePullSecrets (best attached to the ServiceAccount) with a pull-only, rotated credential, or cloud node identity.
  • Secrets in etcd are only encoded unless you configure encryption at rest, ideally with a KMS provider.
  • Audit policies choose how much to record; first match wins, and Secrets belong at Metadata level.
  • The CIS benchmark and kube-bench give a concrete hardening checklist for the API server, kubelet, etcd and workloads.

You now have the security toolbox. Next, learn how to package and deploy applications cleanly, starting with Kustomize.