Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Helm in CI/CD

Upgrades, Rollbacks and Release Strategies

Upgrade Helm releases safely: three-way merges, --atomic and --wait, helm rollback and history, stuck releases, rolling restarts on config change, and a revision stand-in.

Installing a chart once is easy. The real test is the hundredth upgrade, at 2 a.m., when something goes wrong. Helm gives you a revision history and a rollback command, but only if you understand what an upgrade really does. This lesson covers the mechanics, the failure modes and the strategies that keep releases boring.

What an upgrade does

Every helm upgrade creates a new revision of the release and stores it (values, rendered manifest, chart metadata, status) in a Secret. To compute the changes Helm uses a three-way merge between:

  1. the old manifest (revision N),
  2. the live state in the cluster,
  3. the new manifest (revision N+1).

This is why a change someone made with kubectl edit is sometimes kept and sometimes reverted: Helm only patches fields that differ between old and new manifests, and respects live changes to fields neither manifest mentions. It also explains a classic surprise: a field you removed from the chart is removed from the cluster, but a field set manually and never mentioned by the chart is left alone.

Terminal
helm upgrade shop ./webapp -f prod.yaml          # new revision
helm history shop                                # all revisions
helm get values shop --revision 3                # what revision 3 used
helm diff revision shop 3 4                      # (diff plugin) compare two

helm history shows each revision with a status: deployed, superseded, failed, pending-upgrade, uninstalled.

Reusing values: a common trap

By default helm upgrade replaces the previous revision's user-supplied values with only those you pass this time. If you forgot -f prod.yaml, the release silently reverts to chart defaults. Choose one discipline:

  • Always pass the same full set of -f files (best, and declarative).
  • --reuse-values keeps old overrides and merges new --set on top, but ignores new chart defaults, which breaks upgrades that add values.
  • --reset-then-reuse-values resets to the new chart defaults, then applies your previous overrides: usually the safest "I only want to change one thing" option.
Python Stand-in: revisions and the effect of reuse-values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Watch the second line: changing only the tag lost replicas: 5. That single mistake has caused many production incidents.

Making upgrades safe: --wait and --atomic

By default helm upgrade returns as soon as Kubernetes accepts the manifests, not when the Pods are healthy. Add:

Terminal
helm upgrade shop ./webapp --wait --timeout 10m --atomic

--wait blocks until Deployments, StatefulSets, DaemonSets and Services (with load balancers) are ready. --atomic rolls back to the previous revision if anything fails or times out. Pair them with good readiness probes: --wait is only as honest as the probes it relies on. Use --wait-for-jobs when Jobs must finish too.

helm rollback

Terminal
helm rollback shop          # to the previous revision
helm rollback shop 3        # to revision 3
helm rollback shop 3 --wait --cleanup-on-fail

A rollback does not rewind history. It creates a new revision (say 6) whose content equals revision 3. The history stays an honest log:

TEXT
REVISION  STATUS      DESCRIPTION
3         superseded  Upgrade complete
4         superseded  Upgrade complete
5         superseded  Upgrade complete
6         deployed    Rollback to 3

Rollback restores Kubernetes objects, not your data. A database migration run in a pre-upgrade hook is not undone. Two defenses: make migrations backward compatible for one version (expand, then contract), and keep a pre-rollback hook only if you really can reverse the change.

Python Stand-in: rollback creates a new revision

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stuck and failed releases

Symptom Cause Recovery
another operation is in progress CI job killed mid-upgrade; status pending-upgrade helm history, then helm rollback to the last good revision
has no deployed releases The first install failed and left a failed release helm uninstall, then reinstall, or helm upgrade --install with --atomic next time
cannot patch ... field is immutable Changed a Deployment selector or Job spec Delete the object (or release) deliberately and recreate; avoid changing selectors
rendered manifests contain a resource that already exists An object was created outside Helm Adopt it with the ownership annotations or delete it
Release Secrets pile up No history limit --history-max 10

Rollout strategies inside Kubernetes

Helm decides what to apply, Kubernetes decides how Pods roll. Set the strategy in the chart:

YAML
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 0

Also ship a PodDisruptionBudget, correct readiness probes and terminationGracePeriodSeconds. For blue/green or canary releases, use a progressive-delivery controller (Argo Rollouts, Flagger) and let Helm install the Rollout resource.

Restarting Pods when config changes

Changing a ConfigMap does not restart Pods that read it. The standard trick is a checksum annotation on the pod template, so a config change alters the Pod spec and triggers a rollout:

YAML
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
Python Stand-in: checksum annotation triggers a rollout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A safe upgrade checklist

  1. helm diff upgrade and read it.
  2. Back up the data if migrations are not trivially reversible.
  3. Upgrade with --atomic --wait --timeout and a pinned --version.
  4. Run helm test, then watch dashboards for the rollout.
  5. If it fails, helm rollback, then investigate from Pod logs, not guesses.

Practise a rollback on a staging cluster before you need it. The first time you run it should not be during an outage.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does helm rollback shop 3 do to the history?

    History is append-only; the rollback becomes a new revision.
  2. 2. You run helm upgrade --set tag=1.1 without -f prod.yaml. What happens by default?

    Without reuse flags, only the values you pass now are used on top of chart defaults.
  3. 3. What does --atomic imply?

    --atomic sets --wait and rolls back if the upgrade fails.
  4. 4. How can you restart Pods when a ConfigMap in the chart changes?

    A changed annotation changes the Pod spec and triggers a rolling update.

Summary

  • Each upgrade adds a revision and uses a three-way merge; always pass the full values or use --reset-then-reuse-values.
  • Use --wait --atomic --timeout, --history-max, and pinned versions.
  • helm rollback appends a new revision and does not undo data changes.
  • Use checksum annotations to roll Pods on config changes, and practise recovery before an incident.

Next: Project: A Production-Ready Web App Chart