Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 36 ยท Operations and Upgrades

Multicluster Istio: Topologies, East-West Gateways, Failover

Connect several Kubernetes clusters in one Istio mesh: single versus multi network, multi-primary and primary-remote models, shared root CA, east-west gateways, remote secrets and locality failover.

One cluster is a single point of failure for a region, and one cluster cannot always hold everything: compliance may require separate regions, teams may own separate clusters, and large organizations hit scaling limits. A multicluster mesh lets services in different clusters discover each other, talk over mutual TLS and fail over between clusters, as if they were one big mesh.

The product cannot run on this site: commands are read-only and the Python stand-ins model failover and trust checks.

Three decisions

1. Network model. If pods in every cluster can reach each other's pod IPs directly (flat networking, VPN, peering), it is a single network. If not, each cluster is a separate network and cross-cluster traffic must pass through east-west gateways: an Istio gateway in each cluster that accepts traffic from the other clusters (using SNI-based AUTO_PASSTHROUGH so the mTLS session is end to end between the client sidecar and the remote workload).

2. Control plane model.

  • Multi-primary: every cluster runs its own istiod. Each istiod watches the Kubernetes API servers of all clusters to learn endpoints. This is the most resilient: losing one cluster does not stop configuration of the others.
  • Primary-remote: one cluster (primary) runs istiod; remote clusters have no control plane and their proxies connect to the primary's istiod. It is cheaper but the primary is a dependency for the remotes.
  • External control plane: istiod runs outside the data plane clusters entirely, typically a managed offering.

3. Trust. All clusters must trust the same root. Workload certificates in each cluster are signed by an intermediate CA that chains to a shared root CA, so a sidecar in cluster 1 accepts the identity of one in cluster 2. Plug your own CA into each cluster via the cacerts secret in istio-system. Using Istio's self-signed CA separately per cluster breaks cross-cluster mTLS.

flowchart LR
  subgraph c1["cluster1 / network1"]
    I1["istiod"]
    S1["reviews v1"]
    EW1["east-west gateway"]
  end
  subgraph c2["cluster2 / network2"]
    I2["istiod"]
    S2["reviews v2"]
    EW2["east-west gateway"]
  end
  I1 <-->|"remote secrets: watch each other's API server"| I2
  S1 -->|"mTLS via SNI"| EW2 --> S2

Multi-primary on different networks: each istiod sees both clusters; traffic crosses through the east-west gateways.

Setting it up, in outline

Each cluster gets a name, the mesh ID and its network label. In the IstioOperator or Helm values:

YAML
global:
  meshID: shop-mesh
  multiCluster:
    clusterName: cluster1
  network: network1

Label the system namespace with the network (kubectl label ns istio-system topology.istio.io/network=network1), install the east-west gateway in each cluster and expose the services through it. Finally create remote secrets so each istiod can read the other cluster's Kubernetes API:

Terminal
istioctl create-remote-secret --context=cluster2 --name=cluster2 | kubectl apply -f - --context=cluster1
istioctl create-remote-secret --context=cluster1 --name=cluster1 | kubectl apply -f - --context=cluster2

The remote secret holds a service account token with read-only permissions; protect it like any credential and rotate it. Verify with istioctl remote-clusters and by calling a service that exists in both clusters from each side and watching the responses alternate.

Naming and service discovery

A Kubernetes Service named reviews in namespace shop in both clusters becomes one logical service: its endpoints are merged. This is called namespace sameness: a namespace with the same name is treated as the same across clusters, and the same applies to service accounts, which is what makes identity-based authorization work across clusters. Keep names consistent, and keep namespaces with the same name in every cluster owned by the same team.

Locality and failover

With endpoints in several clusters, you do not want every request to cross regions. Locality load balancing prefers endpoints close to the caller: same zone first, then same region, then others. Istio takes the locality from the node labels (topology.kubernetes.io/region and zone). Locality-aware load balancing is on by default but needs outlier detection configured (see Circuit breaking) to know when to fail over, and you can set explicit failover rules:

YAML
trafficPolicy:
  outlierDetection:
    consecutive5xxErrors: 3
    interval: 10s
    baseEjectionTime: 30s
  loadBalancer:
    localityLbSetting:
      enabled: true
      failover:
        - from: europe-west1
          to: us-east1

A distribution setting (distribute) can instead split traffic by percentage between localities.

Cross-cluster failover is not magic: your data tier (databases, caches) must also exist in the failover region, and latency between regions can be tens or hundreds of milliseconds.

Try it: locality failover

Python Which endpoints serve this caller?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: trust root check

Python Do both clusters share a root of trust?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When are east-west gateways required?

    On a single flat network pods can talk directly; separate networks need gateways to carry cross-cluster traffic.
  2. 2. What is the difference between multi-primary and primary-remote?

    Multi-primary is more resilient; primary-remote depends on one control plane.
  3. 3. Why must clusters share a root CA?

    mTLS needs a common root of trust for the workload certificates.
  4. 4. What is namespace sameness?

    Services and service accounts with the same namespace and name are merged across clusters.

Summary

  • Choose network topology (single or multi network), control plane model (multi-primary, primary-remote) and trust (shared root CA).
  • East-west gateways carry cross-network mTLS traffic; remote secrets let istiod discover endpoints across clusters.
  • Namespace sameness merges services with the same name.
  • Locality load balancing and outlier detection provide failover; plan your data tier too.

Next: Project: a canary rollout for Bookinfo