Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 65 ยท Load Balancing and Traffic

Load Balancing Algorithms

Compare round robin, weighted round robin, least connections, least response time, IP hash and power of two choices, and simulate them in Python.

A load balancer must answer one question thousands of times a second: which server gets this request? The rule it uses is the load balancing algorithm. The best choice depends on whether your requests are similar or very different in cost, whether servers are identical, and whether a client must return to the same server.

Algorithms fall into two families. Static algorithms ignore the current state of the servers and follow a fixed rule. Dynamic algorithms look at live information such as active connections or response times. Dynamic ones adapt better to uneven work but need more bookkeeping.

Round robin

Send request 1 to server A, request 2 to B, request 3 to C, request 4 back to A, and so on. It is trivial, needs no state beyond a counter and spreads requests evenly. It works well when requests cost about the same and servers are identical. It does poorly when some requests are very heavy: an unlucky server can receive several heavy ones in a row.

Weighted round robin

Give each server a weight proportional to its capacity. A server with weight 3 gets three times as many requests as one with weight 1. Use this when machines differ (a 16-core server next to an 8-core one) or when you want a gentle canary: weight 1 for the new version against 19 for the old gives it 5% of traffic.

Least connections

Send the request to the server with the fewest active connections. This adapts to uneven requests: servers stuck on slow work have many open connections, so they get fewer new ones, while quick servers get more. It is the usual default for long-lived or variable-cost requests such as uploads, WebSockets and database proxies. Weighted least connections divides each server's connections by its weight.

Least response time

Choose the server with the lowest recent response time (often combined with fewest connections). It routes around a server that is slow for any reason: a noisy neighbor, a garbage-collection pause, a cold cache. The cost is collecting accurate timing data, and the risk of herding: if everyone sees one server as fastest, it becomes overloaded until its numbers worsen.

IP hash and consistent hashing

Compute a hash of the client IP (or another key like user ID) and map it to a server. The same key always reaches the same server, which gives stickiness without a connection table and is useful for caches (the same key hits the same cache node). The downside of simple hashing hash(key) % number_of_servers is that adding or removing a server remaps most keys, which empties caches. The fix is consistent hashing, the next lesson.

Random and power of two choices

Random selection is surprisingly decent for large pools of identical servers and needs no coordination. A much better variant is power of two choices: pick two servers at random and send the request to the one with fewer connections. This tiny change massively reduces the worst imbalance, because a heavily loaded server rarely wins the comparison, and it needs only two lookups rather than scanning every server. Many modern balancers use it.

Pick the algorithm

Round robin Equal servers, similar requests.
Weighted Unequal servers or canary releases.
Least connections Variable or long-lived requests.
Hash Same key, same server. Cache friendly.

Start with round robin or least connections.

1 / 2

Comparing algorithms with a simulation

Theory is easier to trust once you see it. The simulation sends 3,000 requests with very uneven costs (most are light, a few are heavy) to four identical servers and measures how evenly the work is spread, using different algorithms. To keep it simple, each request starts when it arrives and we track the total work and the number of active requests per server at the moment of each decision.

Python Round robin vs least connections vs two choices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two columns matter. Spread is how evenly the total work is divided (smaller is better), and peak backlog is the most unfinished work ever piled onto one server. Random choice is clearly the worst at spreading work. Round robin ignores request cost, so it does reasonably only because its fixed pattern happens to scatter the heavy requests here; change the seed and it can pile several of them on one server. Least connections and two choices react to how busy each server actually is, so they have the smallest peak backlogs, and two choices achieves that while looking at just two servers per request. Exact numbers depend on the seed; the pattern (dynamic beats static for uneven work) is the point.

Weighted round robin in code

Weighted round robin must interleave servers smoothly rather than sending a burst to the heavy server. A common method is smooth weighted round robin: each round, add every server's weight to its running score, pick the highest score, then subtract the total weight from the winner.

Python Smooth weighted round robin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The big server gets 10 of 16 picks, the medium 4 and the small 2, in proportion 5:2:1 and spread out rather than clumped.

Hashing and what happens when the pool changes

A simple hash-based rule maps each key to hash(key) % servers. It gives perfect stickiness, until a server is added or removed. This simulation counts how many of 10,000 keys change server when the pool grows from four to five servers.

Python Modulo hashing remaps most keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About four out of five keys move. For a cache cluster that means most cached data suddenly lives on the wrong node and the database takes the full load. The ideal answer would move only about 1 in 5 keys (the share the new server should own), which is what consistent hashing achieves.

Choosing and tuning

Situation Good choice
Identical servers, uniform short requests Round robin or random
Servers differ in size Weighted round robin / weighted least connections
Variable or long-lived requests Least connections or two choices
One server often slower Least response time (with care)
Need stickiness or cache locality Consistent hashing on user or key
Gradual rollout of a new version Weighted split (canary)
Huge pool, many balancer instances Random or two choices (no shared state)

Whatever you pick, combine it with health checks so failed servers are skipped, and with slow start so a newly added server ramps up instead of receiving its full share while its caches are cold (health checks and failover). Measure the result: per-server request rate, CPU and p99 latency should look similar across the pool.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which algorithm is best when requests vary widely in how long they take?

    Least connections notices servers stuck on slow work and sends new requests to less busy ones.
  2. 2. What does power of two choices do?

    Two random samples plus a comparison avoid the worst imbalance with almost no extra cost.
  3. 3. Why is hash(key) % N a poor choice when servers are added or removed?

    Changing N changes the result for most keys, which is why consistent hashing exists.
  4. 4. When is weighted round robin useful?

    Weights send traffic in proportion to capacity or to a deliberate small share for a new version.
  5. 5. What risk comes with least-response-time balancing?

    If everyone prefers the same fast server it slows down; stale data makes oscillation worse.

Summary

  • Static algorithms (round robin, weighted, hash, random) follow fixed rules; dynamic ones (least connections, least response time, two choices) use live load.
  • Round robin suits uniform requests; least connections or two choices suit uneven work.
  • Weights handle unequal servers and canary releases.
  • Plain modulo hashing remaps most keys when the pool changes; consistent hashing fixes that.
  • Always pair an algorithm with health checks and slow start, and verify balance by measuring.

Next: Consistent Hashing