Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Storage and Scaling

Horizontal Pod Autoscaling in OpenShift

Scale deployments automatically with the Horizontal Pod Autoscaler: CPU and memory targets, the replica formula, requests, min and max, cooldowns, KEDA and the cluster autoscaler.

Traffic is never flat. Running enough pods for the daily peak all day wastes money; running too few makes users wait. The Horizontal Pod Autoscaler (HPA) watches a metric, usually CPU, and changes the replica count of a Deployment to keep it near a target. "Horizontal" means more or fewer pods, in contrast to "vertical" scaling, which gives a pod more CPU or memory.

You cannot run a cluster here. The oc commands and YAML are read-only; the Python stand-ins implement the HPA formula so you can see the behavior.

Prerequisite: resource requests

The HPA calculates utilization as a percentage of the pod's CPU request. Without a request there is nothing to compare to, and the HPA reports <unknown>. Always set requests (limits and quotas):

YAML
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: "1"
    memory: 512Mi

The cluster must also serve metrics: the built-in monitoring stack provides the Metrics API (oc adm top pods works when it does).

Creating an HPA

Terminal
oc autoscale deployment/shop-api --min=2 --max=10 --cpu-percent=70
oc get hpa
oc describe hpa shop-api

The equivalent YAML with the autoscaling/v2 API:

YAML
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shop-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

The formula

Every 15 seconds (by default) the controller computes:

TEXT
desiredReplicas = ceil( currentReplicas * currentMetric / targetMetric )

If 4 pods average 90% CPU and the target is 60%, then ceil(4 * 90 / 60) = 6. The result is clamped between minReplicas and maxReplicas. A small tolerance (10%) avoids flapping when the ratio is near 1.

Python Stand-in: the HPA formula

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Simulating a traffic spike

Scale-up reacts quickly, scale-down waits (the stabilization window, 5 minutes by default) so that a brief dip does not remove capacity just before the next surge.

Python Stand-in: autoscaling over time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Watch how the pod count rises during the spike but returns to the base level only after the traffic has stayed low for several minutes.

Memory, custom and external metrics

CPU is a poor signal for some apps. You can scale on memory (careful: many runtimes do not release memory when load drops, so the HPA may never scale down), on custom metrics from Prometheus through the Custom Metrics Autoscaler Operator, which is the supported KEDA distribution in OpenShift, or on external signals such as queue length. KEDA can even scale to zero and back when events arrive.

YAML
metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"

Cluster-level scaling

More pods need room. When pods are Pending because no node fits them, the Cluster Autoscaler (through MachineAutoscaler objects on MachineSets) adds worker nodes, and removes idle ones. Both layers must be sized together: an HPA maximum above what the node pool can ever supply just produces pending pods. Vertical scaling is available with the Vertical Pod Autoscaler operator, which recommends or sets requests from observed use. Do not let the HPA and VPA both act on CPU for the same workload.

Troubleshooting

  • TARGETS <unknown>/70%: no CPU request, or metrics are unavailable. Check oc adm top pods.
  • HPA never scales up: pods are throttled by a low CPU limit, so measured usage stays low relative to need; or max is already reached.
  • Flapping: widen the scale-down stabilization window.
  • Pods Pending after scale-up: quota exhausted or no node capacity.
  • Stateful apps often cannot scale horizontally without application support.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does CPU utilization in an HPA compare against?

    Utilization is current usage as a percentage of the requested CPU, so requests are mandatory.
  2. 2. Four pods average 90% CPU, target 60%. How many replicas does the formula give?

    ceil(4 * 90 / 60) = 6.
  3. 3. Why does scale-down use a stabilization window?

    Waiting prevents oscillation when traffic fluctuates.
  4. 4. Pods are Pending after the HPA scaled up. Which component adds nodes?

    Pod-level and node-level autoscalers work together.

Summary

  • The HPA adjusts replicas to hold a metric near its target; set CPU requests first.
  • desired = ceil(current * metric / target), clamped by min and max.
  • Scale-down is deliberately slow; tune behavior.
  • Use KEDA or custom metrics for queues and request rates.
  • Cluster Autoscaler provides the nodes the pods need.

Next: Quotas and limit ranges