Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 65 ยท Projects

Project: A Load Balancer Simulator

Build a load balancer simulator in Python: round robin, least connections, weighted choice, health checks and failover, then measure latency and fairness under load.

In this project you build a small discrete-time simulator of a load balancer and the servers behind it. You will implement several balancing algorithms, add health checks and failover, and then measure which algorithm gives the best latency when servers are unequal or when one fails. Everything runs in your browser; no real network is involved, which makes the experiments repeatable.

By the end you will have a harness you can reuse to test any routing idea: add a function, rerun, compare numbers.

What you will build

flowchart LR
    G["Request generator"] --> LB["Load balancer (policy + health)"]
    LB --> S1["Server 1"]
    LB --> S2["Server 2"]
    LB --> S3["Server 3"]
    S1 --> M["Metrics"]
    S2 --> M
    S3 --> M

The generator sends requests; the balancer picks a server; each server has a capacity and a queue.

The model is intentionally small: each server processes one request per time step per unit of its speed, and requests have different sizes (work units). A server's load is the total work waiting in its queue. We record how long each request takes from arrival to completion.

Step 1: servers and requests

Python Servers with queues

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three requests arrive one per step but each needs two steps of work, so waiting time grows. Latency here means "time in queue plus time in service".

Step 2: balancing policies

We implement each policy as a function taking the list of healthy servers and returning one.

Python Four policies and a comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table carefully. Round robin and random treat a fast server and a slow server the same, so the slow ones build queues while the big one idles. Least-loaded uses real information and smooths this out. Power of two choices samples just two servers and takes the less loaded one; it gets most of the benefit without needing a view of every server, which is why it is popular in large systems.

Experiment: change the big server's speed to 1.0 (all equal) and rerun. Round robin should now look much better, because its assumption of equal servers is true.

Step 3: health checks and failover

A balancer must stop sending traffic to a dead server. Real systems use active checks (ping a /health endpoint every few seconds) and passive checks (mark a server bad after several failed requests). Detection is never instant, so requests sent during the gap are lost or retried.

Python Failure, detection delay and recovery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The crash happens at t=21 but the next check runs at t=25, so every request sent to server b in between is lost. After the repair at t=46 the balancer also keeps ignoring b until the check at t=50. Shortening the check interval reduces losses but costs more check traffic; retrying a failed request on another server (when it is safe to repeat, see idempotency in Design a Payment System) hides most remaining errors from users.

Step 4: extensions to try

  1. Weighted round robin: give the big server a weight of 2 and compare with least-loaded.
  2. Sticky sessions: hash a user ID to a server so the same user always lands on the same one. Measure how unevenly load spreads when a few users are very heavy.
  3. Slow start: after recovery, ramp a server's share up over 10 steps rather than flooding it.
  4. Overload shedding: drop requests when every queue exceeds a limit, and measure the p99 that remains.
  5. Retries with a budget: retry failed requests at most 10 percent of the time to avoid a retry storm.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does round robin do poorly when servers have different speeds?

    Equal shares only make sense for equal servers.
  2. 2. What is the advantage of 'power of two choices'?

    It needs only two load readings per request, which scales well.
  3. 3. Why are some requests lost after a server crashes even with health checks?

    Until the next check, the balancer still believes the dead server is healthy.
  4. 4. Which technique prevents a recovered server from being flooded immediately?

    Ramping its share up gradually lets caches warm and avoids a new overload.

Summary

  • A simulator makes load-balancing ideas measurable and repeatable.
  • Random and round robin assume equal servers; least-loaded and power-of-two choices use load information.
  • Health checks trade detection delay against check cost; retries and slow start soften failures.
  • Always compare policies with tail latency (p99), not just the average.

Next: Project: A Sharded Cache with Consistent Hashing