Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 36 ยท Operations and Upgrades

Istio Canary Upgrades with Revisions and Revision Tags

Upgrade Istio safely: in-place versus canary upgrades, installing a second control plane revision, moving namespaces with revision tags, validating, rolling back and removing the old version.

Istio releases new minor versions regularly, and the control plane (istiod) and the sidecars it manages must be kept supported and secure. The upgrade is delicate because every application pod carries a proxy: changing istiod is easy, but proxies only change when pods restart. A good strategy lets you upgrade gradually, test on a few workloads and go back instantly if something breaks. Istio's answer is the canary upgrade with revisions.

The product cannot run on this site: commands are read-only and the Python stand-ins plan and check the steps.

Two upgrade styles

In-place upgrade: you upgrade the one istiod Deployment (istioctl upgrade or helm upgrade). It is simple, but the new control plane immediately controls all existing proxies, and a bug affects everything at once. Rolling back means reinstalling the old version.

Canary upgrade: you install a second control plane next to the first, labeled with a revision name. Namespaces choose which revision injects their sidecars and configures them. You move namespaces one at a time. This is the recommended approach for production.

flowchart TB
  subgraph before["Before"]
    A1["istiod 1-24-0"] --> N1["namespaces: shop, billing, search"]
  end
  subgraph during["During the canary"]
    A2["istiod 1-24-0"] --> N2["billing, search"]
    B2["istiod 1-25-0"] --> N3["shop (canary)"]
  end

Both control planes run at once; each namespace belongs to exactly one revision.

Step by step

  1. Read the release notes and the upgrade notes for removed or changed fields. Run the pre-check (istioctl x precheck) and istioctl analyze -A on the current cluster.
  2. Install the new revision alongside the old one. Use the new version's istioctl, or the Helm istiod chart with --set revision=1-25-0.
Terminal
istioctl install --set revision=1-25-0 -y
kubectl get pods -n istio-system -l app=istiod

Two istiod Deployments exist now, named istiod (default) and istiod-1-25-0. Revision names may not contain dots, so 1.25.0 becomes 1-25-0.

  1. Move a canary namespace by replacing its injection label. A namespace cannot have both istio-injection and istio.io/rev, so remove the old one:
Terminal
kubectl label namespace shop istio-injection- istio.io/rev=1-25-0 --overwrite
kubectl rollout restart deployment -n shop

Pods created after the label change get the new proxy version. Check with istioctl proxy-status, which lists each proxy with the istiod it is connected to and its version.

  1. Validate. Watch error rates, latency and Kiali's health for the canary namespace, run smoke tests and check istioctl analyze -n shop. Leave it running for a while under real traffic.
  2. Roll through the rest of the namespaces, least critical first.
  3. Remove the old control plane once nothing uses it:
Terminal
istioctl uninstall --revision 1-24-0 -y

Delete the old revision only after istioctl proxy-status shows every proxy on the new one.

Revision tags

Relabeling namespaces on every upgrade is tedious and must be repeated for GitOps manifests. Revision tags add a stable alias. You point a tag such as prod-stable at a revision, label namespaces with istio.io/rev=prod-stable once, and later move all of them by re-pointing the tag:

Terminal
istioctl tag set prod-stable --revision 1-24-0
istioctl tag set prod-canary --revision 1-25-0
istioctl tag list

To upgrade the stable group later: istioctl tag set prod-stable --revision 1-25-0, then restart workloads. The tag default is special: it makes a revision the default injector for namespaces labeled istio-injection=enabled.

Gateways and rollbacks

Gateways deployed through injection also pick up a revision: set the istio.io/rev label on the gateway Deployment and restart it. With gateway injection each team can upgrade its own gateways, but the ingress gateway for a canary should ideally be a separate, labeled gateway, so you can send a slice of traffic through it first.

Rollback is the benefit of the canary approach: relabel the namespace back to the old revision and restart the Deployments. No reinstall is needed because the old control plane is still running.

Keep within supported skew

Proxies are supported only when they stay close to the control plane version; the project documents that the data plane may lag the control plane by a limited number of minor versions. Upgrade one minor version at a time, do not leave proxies on an old release for months and check the support table of the exact releases you use. Remember that new pods get the new proxy, old pods keep the old one until restarted.

Try it: plan the migration order

Python Namespace migration plan generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: check proxy versions after a rollout

Python Find proxies left on the old version

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of a canary upgrade over an in-place upgrade?

    Namespaces move one at a time; going back is a label change and restart.
  2. 2. Which label selects a revision for sidecar injection in a namespace?

    istio.io/rev=<revision-or-tag> chooses the control plane; it replaces istio-injection.
  3. 3. Why use a revision tag such as prod-stable?

    Namespaces label the tag; changing the tag's target upgrades them all on the next restart.
  4. 4. When do existing pods get the new proxy version?

    Injection happens at pod creation; running pods keep their proxy until restarted.

Summary

  • Prefer canary upgrades: install a new revision beside the old one.
  • Move namespaces with istio.io/rev (or a tag), restart workloads and validate with telemetry.
  • Verify with istioctl proxy-status, then uninstall the old revision.
  • Upgrade one minor version at a time and keep data plane versions close to the control plane.

Next: Ambient mode: ztunnel and waypoint proxies