Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 37 ยท Communication

Service Discovery and Load Balancing

How services find each other when instances come and go: registries, client-side vs server-side discovery, health checks, round-robin and least-connections balancing.

In a monolith, one part of the program calls another with a function call. In microservices, the orders service must call "the inventory service" over the network, and the question is: at which address? Instances start, stop, crash and scale up and down all day, often with a new IP address each time. A hard-coded address stops working the first time an instance moves.

Service discovery solves this: instances register themselves (or are registered by the platform), and callers look up the current list by name. Load balancing then picks one instance from the list for each request. This lesson explains both and lets you run a registry with health checks and two balancing algorithms.

Finding the right instance

Registry A live list of names and addresses.
Health checks Remove instances that stop answering.
Load balancer Pick one instance per request.
1 / 2

The service registry

A service registry is a database of what is running right now: for each service name, the addresses of its healthy instances. It must be highly available itself, since everything depends on it, so real registries (Consul, etcd, ZooKeeper, Eureka) run as small clusters.

There are two ways instances get into the registry.

  • Self-registration. Each instance tells the registry on startup (inventory at 10.0.0.7:8080) and sends regular heartbeats. If the heartbeats stop, the registry removes it.
  • Platform registration. An orchestrator that starts the instances (Kubernetes, ECS) already knows where they are and maintains the list for you. The service does nothing special.

Either way, a lookup returns a list, not a single address, because several instances usually run at once for capacity and resilience.

Client-side and server-side discovery

In client-side discovery, the caller asks the registry for the list and picks an instance itself, using a load-balancing library. Fewer hops, no central proxy to scale, but every caller needs discovery logic and a library per language.

In server-side discovery, the caller sends the request to a stable address, such as a DNS name or load balancer, and that component looks up instances and forwards the request. The client stays simple. The balancer is an extra hop and an extra thing to keep alive.

Kubernetes is the best-known example of the second style, and the read-only YAML below is real. A Service gives a group of pods one stable name, and the cluster keeps the list of healthy pods behind it:

YAML
apiVersion: v1
kind: Service
metadata:
  name: inventory
spec:
  selector:
    app: inventory
  ports:
    - port: 80
      targetPort: 8080

Any pod in the same namespace can then call http://inventory/stock/sku-1, and the platform spreads the calls across the pods that are ready. The next steps of the topic, containers and orchestration, come in Containers, Docker Compose and Kubernetes. Gateways (API Gateway and Backend for Frontend) usually rely on the same mechanism to find their targets.

Health checks

An address in the list is useless if nothing answers there. A health check is a cheap question the platform asks regularly, usually an HTTP GET /health that returns 200 if the instance can serve traffic. Instances that fail several checks in a row are removed from the list until they recover.

Good health checks distinguish two questions:

  • Liveness: "Is the process stuck and in need of a restart?"
  • Readiness: "Can it handle requests right now?" A service that is still warming up, or has lost its database, is alive but not ready.

Keep checks fast and honest. A check that always returns 200 hides failures, and one that calls five other services can take the whole fleet out of rotation when one dependency has a bad minute.

A registry with health checks

The stand-in below is a registry in plain JavaScript. Instances register with a name and address and report heartbeats with a time stamp; a lookup returns only those whose last heartbeat is recent enough. We use a fake clock so the output is exactly reproducible. Real registries do the same thing with real timers and a network protocol.

JS Registry with heartbeats and expiry

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The silent instance simply fell out of lookups when its time to live ran out, with no one needing to notice or intervene. The same design gives you automatic recovery: when it starts beating again, it returns to the list.

Load balancing algorithms

Given a list of healthy instances, which one gets the next request? The two classic answers:

  • Round-robin. Rotate through the list: first, second, third, first again. Simple and fair when requests cost about the same.
  • Least connections. Send the request to the instance with the fewest requests in progress. Better when requests vary a lot in duration, since a busy instance naturally receives less.

Others include random choice, weighted choices (bigger machines get more traffic) and consistent hashing (the same key always goes to the same instance, useful for caches). A popular simple trick is "power of two choices": pick two instances at random and take the less loaded.

JS Round-robin versus least connections

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Round-robin sent every fourth request to the same instance regardless of how busy it was, so an unlucky instance that received a slow request kept receiving more. Least connections noticed the busy instance and routed around it, which cut the longest wait from 6 ticks to 0 here. When all requests are equal, the two perform about the same, so measure before choosing.

Pitfalls

  • Stale lists. Clients that cache addresses for too long keep calling dead instances. Keep caches short, and combine discovery with timeouts and retries (Timeouts, Retries and Exponential Backoff).
  • Thundering herds. When a fleet restarts, every instance registers and every client refreshes at the same moment. Add small random delays.
  • Optimistic health checks. Passing /health while the database is down sends users to broken instances.
  • Retrying a hopeless instance. Combine balancing with circuit breakers so a failing instance is skipped quickly (Circuit Breakers, Bulkheads and Fallbacks).
  • A single registry node. If it fails, discovery fails. Run a cluster, and let clients keep their last good list as a fallback.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does service discovery solve?

    Instances change address constantly, so callers look up a name and get the current list.
  2. 2. What is the difference between client-side and server-side discovery?

    Client-side puts the lookup and choice in the caller; server-side hides them behind a stable address.
  3. 3. Why use a heartbeat with a time to live?

    When heartbeats stop, the entry expires, so crashed instances disappear from lookups without manual work.
  4. 4. When is least-connections usually better than round-robin?

    Round-robin ignores how busy an instance is; least connections steers new work away from loaded instances.
  5. 5. What does a readiness check tell the platform?

    A process can be alive but not ready (for example still warming up). Readiness controls whether it receives traffic.

Summary

  • Instances come and go, so callers look up services by name in a registry or through the platform instead of hard-coding addresses.
  • Client-side discovery makes the caller choose; server-side discovery hides the choice behind a stable address, as Kubernetes Services do.
  • Heartbeats with a time to live and health checks keep dead instances out of the list; separate liveness from readiness.
  • Round-robin is simple and fair for similar requests; least connections adapts when request times differ.
  • Guard against stale caches, optimistic health checks and a single registry node.

Next: Sagas: Distributed Transactions Without 2PC.