Skip to content
Tutorials Center
My learning

Topics

Lesson 104 of 119 ยท Spring Cloud

Load Balancing and OpenFeign

Spread calls across service instances with Spring Cloud LoadBalancer, and write declarative HTTP clients with OpenFeign and Spring's @HttpExchange interfaces.

Service discovery gives a caller a list: "catalog-service is running at these three addresses right now." Two questions remain. Which of the three should the next request go to? And how do you write the calling code without drowning in URL building, JSON parsing and status-code checks for every endpoint?

The first question is load balancing: spreading requests across instances so none is overloaded, and steering around instances that fail. The second is answered by declarative HTTP clients: you write a Java interface that describes the remote API, and a library generates the implementation. Spring Cloud offers Spring Cloud LoadBalancer for the first and Spring Cloud OpenFeign for the second, and Spring Framework itself now has HTTP interface clients (@HttpExchange), the modern alternative to Feign.

As in the previous lesson, the runnable examples rebuild each mechanism in plain Java with real HTTP servers on 127.0.0.1, because our runner has no Spring Cloud. The real Spring code is shown read-only. You'll even build a working mini Feign with java.lang.reflect.Proxy.

Server-side and client-side load balancing

A load balancer can sit in two places:

Server-side Client-side
Where it runs A separate box or process: NGINX, HAProxy, a cloud load balancer, a Kubernetes Service A library inside each calling service
What the caller sees One stable address The full instance list from discovery
Extra hop Yes No, the caller connects directly
Single point of failure Must be made highly available None, each caller balances for itself
Knowledge of the request Limited (often only TCP or HTTP headers) Full: the caller can retry, prefer zones, stick to an instance
Typical pairing Kubernetes, public traffic Eureka or Consul + Spring Cloud LoadBalancer

Public traffic from browsers always hits a server-side balancer first; browsers can't run your balancing library. Between services, both styles are common. With Eureka you get client-side balancing: every service holds the instance list and picks one per request.

Round robin with real servers

The simplest and most common algorithm is round robin: hand requests to instances in turn, 1, 2, 3, 1, 2, 3. It needs one shared counter. Because many threads call the balancer at once, the counter must be an AtomicInteger, and Math.floorMod keeps the index valid even after the counter overflows past Integer.MAX_VALUE into negative numbers.

Java Round-robin load balancing across three local servers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Seven requests over three instances gave a 3-2-2 split, as fair as it gets. The last line shows the overflow trap: after about two billion requests, % returns a negative index and List.get would throw. Spring Cloud LoadBalancer's own round-robin implementation guards against exactly this.

Choosing an algorithm

Round robin assumes every instance is equally strong and every request equally expensive. When that isn't true, other algorithms do better:

  • Random: pick any instance. No shared state at all, and with many requests it evens out. Spring Cloud LoadBalancer ships it as the alternative to round robin.
  • Weighted round robin: an instance with weight 3 gets three times the traffic of one with weight 1. Useful when machines differ in size, or for a canary release that should get only a small share.
  • Least outstanding requests (least connections): send the request to the instance with the fewest calls in flight. It adapts automatically when one instance is slow, because slow instances accumulate in-flight calls.
  • Sticky sessions: keep sending one user to the same instance. Avoid it when you can; stateless services are easier to scale (Sessions and Cookies shows where state usually lives).

The simulation below sends twelve requests through four strategies. In the least-outstanding run, instance B is slow: each of its requests stays in flight for four "ticks" instead of one.

Java Round robin, random, weighted and least outstanding requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Round robin gave each instance four requests regardless of speed. Random produced a lopsided 5-5-2: it is only fair over many requests, not over a dozen. Weighted round robin produced a 6-4-2 split and, thanks to the "smooth" variant (the one NGINX uses), interleaved the instances instead of sending six A's in a row. Least outstanding requests noticed that B's first call was still running and gave it nothing else, splitting the work between A and C (ties go to the first instance in the list); that adaptivity is why Envoy and many service meshes default to a variant of it.

Failing over to another instance

Discovery data is often a little stale, as the previous lesson showed: an instance may have crashed seconds ago and still be listed. A good client-side balancer retries on the next instance when a connection fails, and remembers bad instances for a while so it doesn't keep hitting them. That memory is a tiny circuit breaker per instance.

Here, catalog-b is listed but stopped. Watch the first request discover the problem and the later requests skip catalog-b during its cool-down:

Java Retry on the next instance and skip unhealthy ones

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Request 2 was meant for catalog-b, failed to connect, and was transparently retried on catalog-c. Request 5 was catalog-b's turn again, but the balancer skipped it without even trying. At request 8 the cool-down was over, so catalog-b got one more chance (still down) and the request fell back to catalog-c. The user never saw an error. Notice the side effect: catalog-c now carries catalog-b's share of the traffic too, so the surviving instances need spare capacity.

Retrying a connection failure is always safe: the request never reached the server. Retrying after a read timeout is only safe for idempotent requests, because the first instance may have done the work (Circuit Breakers and Resilience goes into retry rules in depth).

Spring Cloud LoadBalancer

In Spring, you rarely write any of this. Add spring-cloud-starter-loadbalancer (the Eureka client starter already brings it), mark a RestClient.Builder (or WebClient.Builder) bean @LoadBalanced, and use the service name as the host:

Java
@Configuration
class HttpClientConfig {

    @Bean
    @LoadBalanced
    RestClient.Builder loadBalancedRestClientBuilder() {
        return RestClient.builder();
    }
}

@Service
class CatalogClient {

    private final RestClient rest;

    CatalogClient(RestClient.Builder loadBalancedRestClientBuilder) {
        this.rest = loadBalancedRestClientBuilder.baseUrl("http://catalog-service").build();
    }

    BookDto book(long id) {
        return rest.get().uri("/books/{id}", id).retrieve().body(BookDto.class);
    }
}

The @LoadBalanced marker makes Spring add an interceptor to every client built from that builder. Before each request, the interceptor asks the ServiceInstanceListSupplier (fed by Eureka, Consul or Kubernetes) for catalog-service instances, lets the load balancer choose one, and rewrites http://catalog-service/books/1 into http://10.0.4.17:41873/books/1. In Spring Cloud Gateway the same idea is written lb://catalog-service, as you'll see in API Gateway with Spring Cloud Gateway.

The default algorithm is round robin. To switch a service to random, register a configuration for that client (the configuration class must not be picked up by component scanning, or it applies to every service):

Java
public class RandomForCatalogConfig {

    @Bean
    ReactorLoadBalancer<ServiceInstance> randomLoadBalancer(Environment environment,
                                                            LoadBalancerClientFactory factory) {
        String serviceId = environment.getProperty(LoadBalancerClientFactory.PROPERTY_NAME);
        return new RandomLoadBalancer(
                factory.getLazyProvider(serviceId, ServiceInstanceListSupplier.class), serviceId);
    }
}

@Configuration
@LoadBalancerClient(name = "catalog-service", configuration = RandomForCatalogConfig.class)
class LoadBalancerSetup { }

Other behavior is mostly configuration:

YAML
spring:
  cloud:
    loadbalancer:
      cache:
        ttl: 35s                         # how long the instance list is cached
      configurations: health-check       # or zone-preference, same-instance-preference, ...
      retry:
        enabled: true
        max-retries-on-same-service-instance: 0
        max-retries-on-next-service-instance: 1   # the failover you built above

health-check makes the balancer probe each instance's health endpoint and drop failing ones; zone-preference keeps traffic inside the caller's zone or data center when possible, which saves latency and cross-zone network costs. Retries are applied to GET requests only unless you explicitly allow all methods. Exact options differ slightly between blocking and reactive clients, so check the reference documentation for your release train.

Declarative HTTP clients

Every remote call written by hand repeats the same chores: build the URL, set headers, send, check the status, map errors to exceptions, and turn JSON into an object. A declarative client moves all of that into one generic engine. You only write an interface:

Java
interface CatalogClient {
    @GetMapping("/books/{id}")
    BookDto findById(@PathVariable("id") long id);
}

and the framework creates an object that implements it. How can a library implement an interface it has never seen? With a dynamic proxy: java.lang.reflect.Proxy.newProxyInstance creates a class at runtime that implements any interfaces you name, and sends every method call to one InvocationHandler. The handler reads the method's annotations and arguments and performs the HTTP call. (Reflection and Annotations explain the building blocks.)

Here is a complete mini Feign. It supports @Get with path variables, decodes JSON with Jackson, resolves lb:// URIs through a round-robin balancer, and has an error decoder that turns a 404 into a domain exception:

Java A mini Feign client built with java.lang.reflect.Proxy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The calling code, catalog.findById(1), contains no HTTP at all and gets back a typed Book. Everything else lives in about thirty lines of generic code that work for any interface you annotate. That is all Feign really is, plus many more annotations, encoders, logging and retry hooks.

Spring Cloud OpenFeign

OpenFeign started at Netflix and is now a community project; Spring Cloud OpenFeign integrates it with Spring MVC annotations, Spring's message converters and Spring Cloud LoadBalancer. Add spring-cloud-starter-openfeign, enable scanning, and declare interfaces:

Java
@SpringBootApplication
@EnableFeignClients
public class LoanServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(LoanServiceApplication.class, args);
    }
}

@FeignClient(name = "catalog-service", path = "/books", configuration = CatalogFeignConfig.class)
interface CatalogClient {

    @GetMapping("/{id}")
    BookDto findById(@PathVariable("id") long id);

    @GetMapping
    List<BookDto> search(@RequestParam("q") String query);

    @PostMapping
    BookDto create(@RequestBody NewBookDto book);
}

name = "catalog-service" is a service id, so calls go through discovery and the load balancer automatically. For a fixed external API, add url = "https://api.example.com" instead. You inject CatalogClient like any other bean.

By default, Feign throws a FeignException for any non-2xx status (for example FeignException.NotFound). An ErrorDecoder maps statuses to your own exceptions, just like the switch in the mini Feign:

Java
class CatalogFeignConfig {

    @Bean
    ErrorDecoder catalogErrorDecoder() {
        ErrorDecoder fallback = new ErrorDecoder.Default();
        return (methodKey, response) -> response.status() == 404
                ? new BookNotFoundException("Book not found (" + methodKey + ")")
                : fallback.decode(methodKey, response);
    }
}

Timeouts, logging and circuit-breaker integration are configured per client or for all clients (default):

YAML
spring:
  cloud:
    openfeign:
      client:
        config:
          default:
            connect-timeout: 2000      # milliseconds
            read-timeout: 5000
            logger-level: basic        # none, basic, headers, full
          catalog-service:
            read-timeout: 2000         # this service gets a tighter limit
      circuitbreaker:
        enabled: true                  # wraps calls with Spring Cloud Circuit Breaker

Never rely on Feign's built-in defaults for timeouts: they are long (seconds for connecting, a full minute for reading), far longer than any user will wait. With the circuit breaker enabled, @FeignClient(fallback = CatalogClientFallback.class) names a bean that implements the same interface and answers when calls fail.

HTTP interface clients: the modern alternative

The Spring Cloud team declared OpenFeign feature-complete: it's maintained, but new features go elsewhere. Spring Framework 6 introduced its own declarative clients, HTTP interfaces, and Spring Framework 7 made them easier to register. They need no extra library and run on RestClient (or WebClient for reactive code), so they share its message converters, interceptors and @LoadBalanced support:

Java
@HttpExchange("/books")
interface CatalogClient {

    @GetExchange("/{id}")
    BookDto findById(@PathVariable long id);

    @GetExchange
    List<BookDto> search(@RequestParam("q") String query);

    @PostExchange
    BookDto create(@RequestBody NewBookDto book);
}

@Configuration
class CatalogClientConfig {

    @Bean
    CatalogClient catalogClient(RestClient.Builder loadBalancedRestClientBuilder) {
        RestClient restClient = loadBalancedRestClientBuilder
                .baseUrl("http://catalog-service")
                .defaultStatusHandler(status -> status.value() == 404,
                        (request, response) -> { throw new BookNotFoundException(request.getURI().getPath()); })
                .build();
        return HttpServiceProxyFactory.builderFor(RestClientAdapter.create(restClient))
                .build()
                .createClient(CatalogClient.class);
    }
}

HttpServiceProxyFactory does exactly what your create method did: it builds a proxy whose handler turns annotated methods into HTTP requests. Error mapping happens in the RestClient (defaultStatusHandler), timeouts through its request factory. With many clients, Spring Framework 7's @ImportHttpServices registers whole groups of interfaces at once, and Spring Boot 4 can configure each group's base URL and timeouts from properties.

OpenFeign HTTP interfaces (@HttpExchange)
Comes from Spring Cloud (extra dependency) Spring Framework core
Annotations Spring MVC's @GetMapping, ... @GetExchange, @PostExchange, ...
Underlying client Feign's own client (pluggable) RestClient or WebClient
Reactive return types No Yes, with WebClient
Status Feature-complete, maintained Actively developed
Choose it when Existing code already uses it New code

Design considerations and legacy options

A load balancer is infrastructure you have to think about, because it brings its own choices and failure modes:

  • Health checks decide everything. A balancer is only as good as its idea of "healthy". Use a readiness endpoint that checks what the instance needs to serve (database, cache), not a bare "port is open", and keep it cheap.
  • Stickiness versus statelessness. Session affinity pins a user to one instance, which breaks balance and loses the session when that instance dies. Prefer stateless services with tokens or a shared session store, and use affinity only when you must.
  • Retries multiply load. A client-side balancer that retries on another instance helps with a dead node and hurts when every instance is slow. Combine it with timeouts and a circuit breaker, and retry only idempotent calls.
  • More moving parts. Every balancer layer adds a hop, a config and a thing to monitor. The cure for "complexity" is to use one layer well (for example the platform's, on Kubernetes) rather than stacking client-side, gateway and cloud balancers without a reason.
  • Multi-region deployment. To survive the loss of a whole region, run the service in two or more regions and route users with a global layer (DNS or an anycast load balancer) to the nearest healthy one. Inside a region the instance-level balancing above still applies. Prefer calls that stay in the same zone, and decide up front how data is replicated, because the data usually limits multi-region more than the services do.
  • Lower latency and better use of capacity are the benefits you are paying for: requests spread across instances, the slowest or failed ones are skipped, and capacity can grow by adding instances.

Netflix Ribbon was the original client-side balancer of Spring Cloud and Netflix Zuul its edge gateway, both from the Netflix OSS stack. They are in maintenance mode and no longer part of current Spring Cloud releases: Spring Cloud LoadBalancer replaced Ribbon, and Spring Cloud Gateway replaced Zuul. Old tutorials and codebases (@EnableZuulProxy, ribbon.* properties) refer to the earlier names; the ideas (route table, filters, round robin) carry over directly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A round-robin balancer uses counter.getAndIncrement() % instances.size() as the index. What happens after about two billion requests?

    Java int arithmetic wraps silently. Math.floorMod always returns a value between 0 and size - 1, even for negative counters.
  2. 2. Instance B is twice as slow as A and C. Which strategy automatically sends it fewer requests?

    Slow instances keep more requests in flight, so a least-outstanding-requests balancer picks them less often. Round robin and random ignore speed.
  3. 3. With a @LoadBalanced RestClient.Builder, which base URL should the catalog client use?

    The service id goes where the host would be. The load-balancer interceptor resolves it through discovery for each request.
  4. 4. In the mini Feign, what makes catalog.findById(1) perform an HTTP request although no class implements CatalogClient?

    Dynamic proxies are how Feign and Spring's HttpServiceProxyFactory turn annotated interfaces into working clients.
  5. 5. Your new Spring Boot 4 service needs a declarative client for another internal service. What is the usual recommendation today?

    HTTP interfaces are part of Spring Framework, actively developed, and work with @LoadBalanced builders. OpenFeign remains fine for existing code.

Summary

  • Server-side load balancers (NGINX, cloud balancers, Kubernetes Services) hide instances behind one address; client-side balancers (Spring Cloud LoadBalancer) pick an instance inside the caller using discovery data.
  • Round robin needs an atomic counter and Math.floorMod; random, weighted and least outstanding requests suit uneven instances or requests.
  • Good client-side balancers fail over to the next instance on connection errors and temporarily skip unhealthy ones; retry read timeouts only for idempotent requests.
  • In Spring: mark a RestClient.Builder or WebClient.Builder bean @LoadBalanced and use http://service-name; customize the algorithm with @LoadBalancerClient and behavior with spring.cloud.loadbalancer.*.
  • Declarative clients are dynamic proxies that turn annotated interface methods into HTTP calls, JSON decoding and error mapping.
  • OpenFeign: @EnableFeignClients, @FeignClient(name = "service-id"), Spring MVC annotations, ErrorDecoder, explicit timeouts. HTTP interfaces (@HttpExchange + HttpServiceProxyFactory) are the modern, built-in choice for new code.

Next: API Gateway with Spring Cloud Gateway