Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 79 ยท Application Layer: Names and Web

Proxies, Load Balancers and CDNs

Understand forward and reverse proxies, Layer 4 versus Layer 7 load balancing, balancing algorithms, health checks, CDN edge caching and anycast, with live Node demos.

So far, a web request looked like a direct conversation between a browser and a server. On the real Internet there are usually several machines in the middle. A company may force all employee traffic through a proxy. A busy site puts a load balancer in front of dozens of identical servers. And almost every large site sits behind a CDN, a worldwide network of servers that answer from nearby so that the request never travels to the origin.

These middle boxes all share one idea: they speak the client's protocol on one side and make another connection on the other side. In this lesson you learn what each kind does, how load balancers decide where to send traffic, how they notice failures, and how a CDN uses caching and routing tricks to bring content close to users. You will also build a small working reverse proxy.

Forward proxies and reverse proxies

A proxy is a server that makes requests on behalf of someone else. The two main kinds differ by whom they work for.

flowchart LR
    subgraph L["Forward proxy: works for the clients"]
      C1["Client"] --> FP["Forward proxy"] --> I1["Internet sites"]
    end
    subgraph R["Reverse proxy: works for the servers"]
      I2["Clients on the Internet"] --> RP["Reverse proxy"] --> S1["Server 1"]
      RP --> S2["Server 2"]
    end

A forward proxy hides or controls clients. A reverse proxy hides and protects servers.

A forward proxy sits near the clients, and the clients are configured to use it (or a network forces them through, which is called a transparent proxy). It can:

  • filter content and enforce company policy,
  • cache popular downloads for the whole organization,
  • hide the clients' real IP addresses from the sites they visit,
  • log and authenticate users.

For HTTPS, a forward proxy usually cannot read the traffic. The browser sends CONNECT example.org:443 and the proxy simply forwards the encrypted bytes through a tunnel. Inspecting HTTPS requires the proxy to impersonate sites with its own certificate, which clients must be set up to trust (see TLS and HTTPS).

A reverse proxy sits in front of one or more servers, and clients do not even know it is there. They connect to the public name, and the proxy forwards the request inward. It can:

  • terminate TLS, so the servers behind it speak plain HTTP on a private network,
  • compress and cache responses,
  • route different URLs to different back-end services,
  • hide the real servers, absorb attacks and add a security layer,
  • spread requests over many servers (load balancing).

Because the real client is now the proxy, the server would see only the proxy's address. Proxies therefore add headers such as X-Forwarded-For (the original client IP), X-Forwarded-Proto and the standard Forwarded and Via headers. Applications behind a proxy must trust these headers only from their own proxy, since anyone can send a fake one.

Load balancing

A load balancer is a reverse proxy (or a packet forwarder) whose main job is to distribute traffic over a pool of servers. It gives you three things: more total capacity (scale out by adding servers), higher availability (if a server dies, the others keep serving), and the ability to deploy without downtime (take one server out at a time).

Layer 4 versus Layer 7

Load balancers work at different layers of the layered model.

Layer 4 (transport) Layer 7 (application)
Sees IP addresses, TCP/UDP ports Full HTTP request: URL, headers, cookies
Decision Per connection Per request
Speed Very fast, little state Slower: must parse and often terminate TLS
Can do Spread connections, any TCP/UDP protocol Route /api and /images to different pools, rewrite headers, cache, sticky sessions by cookie
Examples Linux IPVS, cloud network load balancers Nginx, HAProxy (HTTP mode), Envoy, cloud application load balancers

A layer 4 balancer picks a server when a connection starts and forwards every packet of that connection to it (often using NAT or tunneling). It cannot tell a request for /logo.png from one for /checkout. A layer 7 balancer reads each request and can make smarter choices, at the cost of more CPU and the need to hold TLS keys.

Algorithms

How does the balancer choose a server?

  • Round robin: take servers in turn. Simple and fair when requests cost about the same.
  • Weighted round robin: bigger machines get a larger share.
  • Least connections: send to the server with the fewest active requests. Better when some requests take much longer than others.
  • IP hash / consistent hashing: hash the client address (or a key such as a URL) to choose the server, so the same client or object usually lands on the same server. Consistent hashing keeps most assignments stable when servers join or leave, which is why caches use it.
  • Random with two choices: pick two servers at random and use the less loaded one. Surprisingly effective at large scale.

Sticky sessions (session affinity) keep a user on one server, using a cookie or an IP hash. They are a crutch for servers that keep session data in memory; the better design stores sessions in a shared store so any server can answer.

The next demo compares three algorithms on the same stream of 300 requests, where one request in ten is expensive (80 ms) and the rest are cheap (5 ms). It is a virtual-time simulation, so the numbers are identical on every run.

Node.js Round robin versus least busy versus IP hash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The "least busy" strategy keeps the worst wait much lower because it avoids queuing a cheap request behind an expensive one. Round robin and hashing cannot see that.

Health checks

A balancer must stop sending traffic to a dead server. It does this with health checks:

  • Passive: watch real traffic. If a server refuses connections, times out or returns many 5xx errors, mark it down for a while.
  • Active: periodically probe each server, for example GET /health every few seconds. A server that fails a few probes in a row is removed, and it is added back after some successes.

A good health endpoint checks what really matters (can the app reach its database?) without being so strict that one slow dependency takes every server out at once.

Time to build one. This program starts three back-end servers and a reverse proxy on loopback. The proxy rotates over the healthy servers, adds X-Forwarded-For and Via, and an active health check removes a server that has crashed.

Node.js A reverse proxy with round robin and health checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice that app-B takes its turn in the first round, disappears after the check, and the remaining two share the traffic evenly. A production proxy such as Nginx or HAProxy does the same with retries, timeouts, connection pooling and many more options.

Content delivery networks

A CDN is a reverse proxy and cache at global scale. The provider runs hundreds of edge servers (points of presence) in cities around the world. A site points its name at the CDN, which answers requests from the nearest edge. The physical reason is simple: a round trip from Tokyo to a server in Virginia takes well over 100 ms just for light in fiber, and a page needs several such trips. An edge server 10 ms away removes most of that.

flowchart LR
    U["User in Madrid"] --> E["Edge server<br/>in Madrid"]
    E -- "hit: answers directly" --> U
    E -- "miss" --> M["Mid-tier / shield cache"] --> O["Origin in the US"]
    O -. "response, cached on the way back" .-> E

A cache hit stays at the edge. Only misses go to the origin, and often via a shared mid-tier cache so the origin sees each object once.

What a CDN does:

  • Caches static content (images, scripts, video segments, downloads) using the same Cache-Control rules as any HTTP cache. The share of requests answered from cache is the hit ratio.
  • Terminates TLS and HTTP/2/HTTP/3 near the user. The slow handshake round trips happen over a short distance, and the CDN reuses long-lived connections to the origin (see HTTP/1.1, HTTP/2 and HTTP/3).
  • Absorbs attacks. A distributed network can swallow volumetric DDoS traffic that would flatten a single origin, and it can run a web application firewall.
  • Optimizes content: compression, image resizing, minification.
  • Runs code at the edge (serverless functions) for redirects, personalization and A/B tests.

You can purge cached objects when content changes, or use versioned URLs so you never need to.

The simulation below shows why caching at the edge pays off even with a small cache. Requests follow a skewed popularity: a few objects are requested often (like real sites), and the cache keeps the most recently used ones.

Node.js Edge cache hit ratio with LRU eviction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Storing only a few percent of the catalogue already answers a large share of requests, and each hit saves a trip to the origin.

How users reach the nearest edge

How does the user's request land on the nearest edge server? Two techniques are common, often combined:

  • DNS-based steering. The CDN's authoritative DNS server returns a different IP address depending on where the resolver asking is, usually a nearby edge. A CNAME from your site's name to the CDN's name hands the decision to the CDN.
  • Anycast. Many servers in different cities announce the same IP address through BGP. Internet routing then delivers each packet to the topologically closest announcer. If one site fails, it stops announcing and traffic flows to the next-closest. Anycast is also how public DNS resolvers and root servers are spread worldwide, and it helps absorb DDoS attacks by spreading them across all sites.

Anycast works well for short, request-response traffic. For a long TCP connection it relies on routing staying stable for its duration, which is mostly true in practice.

A CDN or proxy changes who can see your data. A CDN that terminates TLS sees all traffic in plain form, and your origin sees only the CDN's addresses. Restrict your origin to accept connections only from the CDN, and read the real client address from the header the CDN documents.

What CDNs are used for, and where they fall short

Beyond static website acceleration, CDNs are used for software and game downloads (a huge patch is served from the edge nearest each user), video streaming, e-commerce sites during sales peaks, APIs and microservices (caching responses and terminating TLS close to the user), and DDoS absorption, because the traffic is spread over many edge locations. The benefits are lower latency, a better user experience, less load and bandwidth cost at the origin, and extra availability when the origin has problems.

The limits are real. Cache inconsistency can show visitors an old version of a page until the cached copy expires or is purged. Dynamic and personalized content cannot be cached widely. A wrong configuration can cache private pages and serve them to the wrong person, so any response that depends on the user must send Cache-Control: private or no-store. Using a CDN also adds a third party in the path, costs money for traffic and features, and makes a CDN outage hit many sites at once.

Python Cache hit ratio and the effect of a stale copy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A longer TTL raises the hit ratio and lowers origin load, but it also lengthens the time that visitors can see out-of-date content. Purging the cache when you publish is how sites keep both.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which statement best describes a reverse proxy?

    A reverse proxy works for the servers: clients talk to it as if it were the site, and it forwards requests inward. A forward proxy works for clients.
  2. 2. A load balancer must send /api requests to one pool and /images requests to another. What does it need?

    Only a balancer that reads the HTTP request can see the URL path. A layer 4 balancer sees addresses and ports.
  3. 3. Requests vary a lot in cost. Which algorithm usually keeps latency lowest?

    Least connections adapts to actual load, so a cheap request is not queued behind an expensive one on a busy server. Round robin and hashing ignore load.
  4. 4. What does an active health check do?

    Active checks send their own probe requests. Passive checks watch real traffic.
  5. 5. How does anycast bring a user to a nearby CDN site?

    With anycast, one address exists in many places, and ordinary routing delivers packets to the nearest announcer.

Summary

  • A forward proxy represents clients (filtering, caching, anonymity); a reverse proxy represents servers (TLS termination, routing, caching, protection).
  • Proxies hide the real client address, so they add X-Forwarded-For, Via or Forwarded, which back ends must trust only from their own proxy.
  • Load balancers spread traffic for capacity and availability. Layer 4 balances connections quickly; layer 7 reads HTTP and can route by URL or cookie.
  • Common algorithms: round robin, weighted, least connections, hashing and consistent hashing. Active and passive health checks remove failing servers.
  • A CDN is a global reverse-proxy cache: nearby edges answer hits, misses go to the origin, and the hit ratio decides the benefit.
  • Users are steered to the nearest edge with DNS answers and BGP anycast.

Next, move beyond request-response and learn how servers push data to browsers in WebSockets, SSE and Real-Time Communication.