Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 39 ยท Load Balancing and Gateways

gRPC Load Balancing

Learn why gRPC needs special load balancing, compare proxy and client-side approaches, round robin, pick_first, weighted policies and name resolution, with runnable Go simulations.

With ordinary HTTP/1.1 APIs, load balancing is nearly automatic: each request opens or reuses a connection, and a load balancer spreads connections across servers. Because requests are short and connections are plentiful, the load ends up fairly even.

gRPC breaks that assumption. It uses HTTP/2, which multiplexes many calls over one long-lived connection. A connection-level (layer 4) load balancer picks a server once, when the connection opens, and then every call from that client goes to the same backend for hours. Ten clients and ten servers can easily end up with one server doing most of the work. You need to balance calls, not connections.

grpc-go cannot run on this site's runner. Real configuration is shown read-only; the runnable blocks simulate the policies with plain Go so you can compare how they spread the load.

Two approaches

flowchart LR
    subgraph P["Proxy (layer 7) balancing"]
        C1["Client"] --> LB["Envoy / nginx / cloud LB"]
        LB --> S1["Server 1"]
        LB --> S2["Server 2"]
        LB --> S3["Server 3"]
    end
    subgraph K["Client-side balancing"]
        C2["Client with<br/>balancer policy"] --> T1["Server 1"]
        C2 --> T2["Server 2"]
        C2 --> T3["Server 3"]
    end

Left: the proxy understands HTTP/2 and spreads individual calls. Right: the client knows all the backends and chooses one per call.

Proxy (L7) Client-side
Extra hop Yes (added latency) No
Client complexity Low, any client Needs resolver and policy support
Moving parts A proxy fleet to run None
Typical tools Envoy, nginx, HAProxy, cloud gRPC load balancers, service meshes grpc-go round_robin, xDS
Best when Untrusted or many different clients, mobile apps Internal services, latency sensitive paths

A layer 4 balancer is fine only if each client opens many connections, or if you accept unevenness. Most cloud load balancers have a gRPC-aware (HTTP/2, layer 7) mode: use it.

Client-side balancing, step by step

Client-side balancing has two jobs:

  1. Name resolution turns the target name into a list of addresses. grpc-go supports dns:///users.internal:50051, passthrough, unix: and custom resolvers (Consul, Kubernetes endpoints, xDS).
  2. A balancing policy decides which address gets each call. It also owns connection management: one sub-connection per backend.

Built-in policies in grpc-go:

  • pick_first (default): connect to the first address that works and use only that one. No balancing.
  • round_robin: rotate across all healthy addresses.
  • weighted_round_robin: round robin that uses server-reported load ("ORCA" metrics) to give faster servers more calls.
  • least_request, ring_hash (sticky by a key, useful for caches), and others, mostly via xDS.

Selecting one in Go:

Go
conn, err := grpc.NewClient("dns:///users.internal:50051",
    grpc.WithTransportCredentials(creds),
    grpc.WithDefaultServiceConfig(`{"loadBalancingConfig": [{"round_robin": {}}]}`),
)

DNS resolution only re-resolves when a connection breaks (or on a timer in some versions), so a new pod may not be noticed quickly. In Kubernetes use a headless service (clusterIP: None) so DNS returns every pod IP rather than one virtual IP, which would defeat client-side balancing. Servers can also send GOAWAY after a maximum connection age (MaxConnectionAge in the server's keepalive parameters) to make clients reconnect and re-resolve.

Simulation: why pick_first is unbalanced

Go Round robin versus pick_first

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

pick_first sends all 1000 calls to the first server. Round robin spreads them evenly.

Smarter policies

Round robin treats all servers as equal and all calls as equal. Real systems have slow servers and heavy calls. Two common improvements:

  • Least requests ("power of two choices"): pick two random backends and send the call to the one with fewer calls in flight. It is simple and avoids slow servers.
  • Weighted: backends report CPU or queue length and get proportionally more or fewer calls.
Go Power of two choices beats random

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Health and ejection

A balancer should stop sending calls to backends that are down. Mechanisms:

  • The connection itself: if a sub-connection breaks, the policy stops using it and retries with backoff.
  • Client-side health checking: the client calls the standard health service of every backend (see health checking) and drops the ones reporting NOT_SERVING.
  • Outlier detection (xDS): eject a backend after too many consecutive errors.
  • Retries across backends (retry policy): a retry can land on a different server.

Graceful shutdown matters

When you deploy, servers go away. Call srv.GracefulStop(): it stops accepting new calls, sends GOAWAY so clients move to other backends, and waits for in-flight calls. Mark the server NOT_SERVING first, wait a few seconds for balancers to notice, then stop. Without this, every deploy produces a burst of UNAVAILABLE errors.

Practical advice

  • Inside a mesh or with xDS, use it: the control plane gives you balancing, retries and outlier detection together.
  • On Kubernetes without a mesh: headless service, round_robin, MaxConnectionAge of a few minutes with grace.
  • For public APIs: put a gRPC-aware (HTTP/2) load balancer in front.
  • Always test balance: look at per-backend request counters after a load test.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is plain connection-level (L4) load balancing a poor fit for gRPC?

    The balancer chooses once per connection; with long-lived multiplexed connections the load becomes uneven.
  2. 2. Which grpc-go policy is the default and does no balancing?

    pick_first connects to the first working address and sends everything there.
  3. 3. In Kubernetes, why use a headless service with client-side balancing?

    A normal ClusterIP hides the pods behind a single address, so the client sees only one backend.
  4. 4. What is the benefit of power-of-two-choices?

    Picking the less loaded of two random backends gives most of the benefit of full least-loaded balancing at very low cost.

Summary

  • gRPC calls share long-lived HTTP/2 connections, so balance calls (layer 7) rather than connections.
  • Use a gRPC-aware proxy or client-side balancing with a resolver and a policy such as round_robin.
  • pick_first is the default and does not balance; weighted and least-request policies handle uneven servers.
  • Combine balancing with health checks, retries, max connection age and graceful shutdown.
  • On Kubernetes use headless services for client-side balancing, or a mesh/xDS.

Next: Health Checking and Reflection