Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 35 ยท Kubernetes

Scaling and Upgrading DigitalOcean Kubernetes

Scale a DOKS cluster at two levels: pods with the Horizontal Pod Autoscaler and nodes with node pool autoscaling, then upgrade Kubernetes safely with surge upgrades.

Traffic changes during the day, so a fixed number of pods and nodes either wastes money or falls over. DOKS scales on two levels. The Horizontal Pod Autoscaler (HPA) changes the number of pods in a Deployment. Node pool autoscaling changes the number of Droplets so those pods have somewhere to run. The two work together: HPA asks for more pods, pods go Pending when nodes are full, and the cluster autoscaler adds a node.

The last part covers upgrades: Kubernetes releases a new minor version about every four months, and old ones lose support. Read-only commands appear as text; the Python blocks are stand-ins for the math and the ordering.

Level 1: the Horizontal Pod Autoscaler

HPA needs the Metrics Server, which DOKS installs by default, and pods with CPU requests (utilisation is measured against the request).

Terminal
kubectl autoscale deployment shop-web --cpu-percent=60 --min=3 --max=12
kubectl get hpa

The same thing as YAML:

YAML
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop-web
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: shop-web }
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }

The controller's formula is simple: desired = ceil(current * currentMetric / targetMetric), clamped to the min and max. It also waits before scaling down (a stabilisation window of about five minutes) to avoid flapping.

Python Stand-in: the HPA formula

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This stand-in scales down immediately, which the real HPA delays. Notice how a single reading of 200% jumps straight to the maximum when it would exceed it: HPA reacts to the ratio, not in tiny steps.

Level 2: node pool autoscaling

Enable it when you create or update a pool:

Terminal
doctl kubernetes cluster node-pool update shop-cluster web \
  --auto-scale --min-nodes 2 --max-nodes 6

The autoscaler adds a node when pods are Pending because of insufficient resources, and removes a node when it has been underused for a while and its pods fit elsewhere. Three caveats:

  • It reacts to requests, not actual usage. Pods with no requests never trigger a scale-up.
  • Scale-up takes a few minutes, because a new Droplet has to boot and join the cluster. Keep minReplicas and min-nodes high enough for sudden spikes.
  • Pods using local state, or protected by strict disruption budgets, can block a scale-down.
Python Stand-in: nodes needed for pending pods

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The simplification here is packing perfectly; real clusters waste some capacity. The HI cap protects your bill: without a maximum, a bug that spawns pods would also spawn Droplets.

Upgrading Kubernetes

Check what is available and upgrade:

Terminal
doctl kubernetes options versions
doctl kubernetes cluster upgrade shop-cluster --version latest

Key facts:

  • You can only move one minor version at a time (for example 1.31 to 1.32).
  • DigitalOcean can apply automatic patch upgrades in a maintenance window you choose (--auto-upgrade, --maintenance-window).
  • A surge upgrade (on by default) creates a new node with the new version before draining an old one, so capacity never dips below what you had. Turn it off only if you cannot afford the extra Droplet briefly.
  • During the upgrade each node is cordoned and drained, pods are rescheduled, and the node is replaced. This is why you want at least two replicas and a readiness probe on every Deployment.
  • Add a PodDisruptionBudget to stop a drain from evicting too many replicas at once.
YAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: shop-web
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: shop-web }
Python Stand-in: surge upgrade order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Test upgrades on a staging cluster first, read the Kubernetes release notes for removed APIs, and take backups of any persistent data before you start.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the HPA scale?

    HPA changes replicas; node pool autoscaling changes the number of nodes.
  2. 2. Current: 4 pods at 90% CPU, target 60%. What does the HPA formula give?

    ceil(4 * 90 / 60) = 6 pods.
  3. 3. Why might the node autoscaler not add a node for a busy pod?

    The autoscaler reacts to pending pods and requests, not real usage.
  4. 4. What does a surge upgrade do?

    Surge keeps capacity steady by creating the replacement before removing the old node.

Summary

  • HPA scales pods using ceil(current * metric / target); the node pool autoscaler scales Droplets so pods can be scheduled.
  • Always set requests, minimums and a maximum so scaling is both effective and affordable.
  • Upgrade one minor version at a time; surge upgrades, replicas, readiness probes and PodDisruptionBudgets keep the app up.
  • Choose a maintenance window and test upgrades on staging first.

Next: DigitalOcean Load Balancers