Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 79 ยท Routing

WAN Technologies: MPLS, Leased Lines and SD-WAN

Learn how companies connect distant sites: leased lines, Frame Relay and ATM history, MPLS label switching and VPN services, and SD-WAN path selection, with Python models.

A company with offices in Berlin, Chicago and Singapore faces a problem the LAN lessons never had: the distance. Nobody can run a private cable across oceans, so the company buys connectivity from carriers, the wide area network (WAN) providers. The way this is done has changed several times over the last fifty years, from dedicated copper circuits to virtual circuits, then to label-switched private networks, and now to software that blends several cheap Internet links into one reliable network.

This lesson follows that story and explains how each generation works: leased lines, the virtual-circuit era of X.25, Frame Relay and ATM, MPLS with its label switching and VPN services, and SD-WAN. You will run a model of a packet crossing an MPLS core, and one of an SD-WAN controller choosing a link for each type of traffic.

Leased lines

A leased line is a dedicated point-to-point digital circuit between two sites, rented from a carrier for a monthly fee. The capacity is reserved for you at all times, whether you use it or not, which makes the service predictable and expensive. Classic rates are the T1 (1.544 Mbit/s) and E1 (2.048 Mbit/s) copper lines, the T3 (about 45 Mbit/s), and the optical SONET/SDH hierarchy (OC-3 at 155 Mbit/s and up). Today the same idea survives as Ethernet circuits over fiber ("Metro Ethernet" and dedicated Internet access).

A leased line has no sharing and no waiting: it is the circuit-switching idea you saw in How Data Travels turned into a product. Its weakness is that connecting n sites with a full mesh of lines needs n(n-1)/2 circuits, so most organizations used a hub-and-spoke layout, with each branch connected to a head office.

The virtual-circuit era: X.25, Frame Relay, ATM

Carriers soon realized that bursty data traffic wasted dedicated circuits. They built shared packet networks in which customers received virtual circuits: a path that exists as an entry in the switches' tables rather than as dedicated wire.

  • X.25 (1970s) provided reliable virtual circuits over unreliable analog lines, with error checking at every hop. It was slow because of this overhead, and was replaced when lines became reliable.
  • Frame Relay (1990s) dropped the hop-by-hop checks. Each frame carries a DLCI, a circuit number that the switches map to the next hop, and customers bought a committed information rate (CIR), the bandwidth the carrier promised, with the right to burst above it if capacity allowed.
  • ATM (Asynchronous Transfer Mode) cut everything into fixed 53-byte cells (5 bytes of header, 48 bytes of payload). The small, constant size gave predictable delay for voice and video and made switches easy to build in hardware, but wasted about 10% of the bandwidth on headers and fitted IP traffic badly.

All three share one idea that matters for what follows: forward by a short label that is swapped at each hop, not by a long IP address. MPLS took that idea, kept it, and made it work with IP.

MPLS: label switching

Multiprotocol Label Switching sits between layer 2 and layer 3: a 4-byte label is inserted in front of the IP packet (it is sometimes called "layer 2.5"). Inside the carrier's network, routers forward the packet by looking only at the label, with a fast exact-match lookup in a small table, instead of a longest-prefix match on the IP address.

Parts of the system.

  • A label has 20 bits for the label value, 3 bits for a traffic class, 1 bit marking the bottom of the label stack, and 8 bits of TTL.
  • A label-switching router (LSR) forwards packets based on labels. The routers at the edge of the MPLS network are label edge routers (LERs), often called PE (provider edge) routers. Routers in the core are P routers.
  • A forwarding equivalence class (FEC) is a group of packets that are treated the same way, for example all packets to one destination prefix or all voice packets.
  • A label-switched path (LSP) is the one-way path that packets of a FEC follow through the network.

The three label operations. The ingress LER classifies the packet and pushes a label. Each core LSR looks up the incoming label in its label forwarding table, swaps it for the label the next router expects and forwards the packet. At the end, the label is popped. In practice the second-to-last router pops the label (penultimate hop popping) so that the egress router does not have to do two lookups.

Labels are distributed between routers by a protocol: LDP (Label Distribution Protocol) builds LSPs along the paths that IP routing chose, and RSVP-TE sets up explicitly engineered paths with reserved bandwidth. Because the route can be chosen by the operator, MPLS allows traffic engineering: sending some traffic over an under-used path, keeping voice on the low-latency route, or reserving a backup path with fast reroute that recovers in about 50 ms after a link failure.

flowchart LR
    CE1["Customer site 1"] --> PE1["PE1: push labels"]
    PE1 -->|"labels 100, 5001"| P1["P1: swap 100 to 200"]
    P1 -->|"labels 200, 5001"| P2["P2: pop outer label"]
    P2 -->|"label 5001"| PE2["PE2: pop VPN label"]
    PE2 --> CE2["Customer site 2"]

A packet crosses an MPLS core with a stack of two labels: an outer transport label that changes at each hop and an inner VPN label that identifies the customer.

MPLS VPN services

The labels allow a carrier to run many customers on one physical network and keep them separate:

  • Layer 3 VPN (L3VPN). Each customer gets a private routing table, a VRF, on the PE routers. The PE routers exchange the customer routes with BGP, and an inner VPN label identifies the right VRF at the egress. Two customers can even use the same private addresses (such as 10.0.0.0/8) without conflict. The customer's router just sees the carrier as a router connected to all its sites.
  • Layer 2 VPN (L2VPN, VPLS, EVPN). The carrier network behaves like a giant Ethernet switch or a virtual wire between sites. The customer runs its own routing and may use any layer-3 protocol.

An MPLS VPN is private because the carrier keeps customer traffic apart, but the traffic is not encrypted. It rides the carrier's infrastructure, and the privacy relies on trust in the carrier and the isolation of labels. A customer who needs confidentiality adds IPsec or another VPN on top.

A packet crossing an MPLS core

The program models the network of the diagram: PE1 classifies packets by destination prefix with a longest-prefix lookup, pushes a stack of two labels, and each core router performs only a label lookup. The second FEC takes a different, engineered path through router P3.

Python Label push, swap and pop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first packet shows the full cycle: PE1 pushes the transport label 100 and the VPN label 5001, P1 swaps 100 for 200, P2 pops the outer label (penultimate hop popping), and PE2 pops the VPN label and delivers the packet to the customer. The second destination follows the engineered path through P3. Notice that the core routers never examine the IP header, and that each label adds 4 bytes, so a 1500-byte payload with two labels needs a 1508-byte link MTU. The MTU of MPLS links is therefore usually raised.

SD-WAN

MPLS is reliable and gives guarantees, but it is expensive, and slow to provision, and it only reaches where the carrier has a presence. Meanwhile ordinary broadband and mobile connections became fast and cheap. SD-WAN (software-defined WAN) uses them together.

An SD-WAN branch device connects to several links at once: MPLS if the company still has it, broadband, LTE or 5G. Between sites it builds encrypted tunnels (IPsec or similar) over each link, an overlay network that hides which physical links carry the traffic. A central controller defines policy, and each device:

  1. Measures every path continuously with probes: latency, loss and jitter.
  2. Classifies traffic by application, for example voice, video conferencing, business apps and backups.
  3. Chooses a path per application according to the policy, and re-evaluates it all the time. If the preferred path degrades, new flows (and sometimes existing ones) move to a better path within a second or so.

The effect is that cheap links can carry most traffic, and the expensive guaranteed link is used only for what needs it, or as a backup. The same overlay also gives easy cloud access: a branch can go directly to a SaaS provider over broadband instead of sending all traffic to the head office first.

Python An SD-WAN controller chooses links per application

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At time 0, broadband meets every need and is the cheapest, so all classes use it. When it gets congested at time 1, voice moves to the premium MPLS link (it has the jitter budget), video moves to LTE (cheap and good enough), and backup traffic stays on broadband, because it tolerates almost anything. When MPLS fails at time 2, no link meets the voice policy, so the controller picks the least bad one and flags it as degraded. When broadband recovers, everything returns to the cheapest link. This mirrors what real products do, with hysteresis added so that traffic does not flap between links.

Comparing WAN options

Technology Era Forwarding Strength Weakness
Leased line (T1, E1, SONET, Ethernet circuit) 1960s to today Dedicated circuit Predictable and private Costly, fixed capacity
X.25 1970s Virtual circuits, hop-by-hop recovery Works on poor lines Slow
Frame Relay 1990s DLCI virtual circuits Cheaper than leased lines Replaced by IP-based services
ATM 1990s 53-byte cells Predictable delay Cell tax, poor fit for IP
MPLS (with L3VPN) 2000s to today Label switching QoS, traffic engineering, private VPNs Expensive, carrier-dependent
Internet VPN (IPsec) 2000s to today IP over the public Internet Cheap No guarantees
SD-WAN 2010s to today Overlay over several links Cost, agility, cloud access Depends on link quality and the vendor

VPN versus MPLS

Both a VPN and an MPLS service connect a company's sites privately, but they do it very differently.

Aspect Internet VPN (IPsec, WireGuard) MPLS VPN service
Path Over the public Internet Over the provider's private backbone
Privacy By encryption of the traffic By logical separation of customers; usually not encrypted
Quality of service Best effort (the Internet gives no guarantee) Classes of service with delay and loss guarantees under an SLA
Cost Low: ordinary broadband is enough High: the carrier charges for the private circuit
Setup Fast: a few hours, any place with Internet Slow: weeks, requires carrier presence at each site
Weak points Unpredictable performance, depends on the Internet Costly, locked to one provider, little flexibility, and no secrecy unless you add encryption
Typical use Remote users, small branches, backup paths Voice and critical applications between large sites

Many organizations combine them: MPLS for critical sites and a VPN over the Internet as a backup, which is the idea behind SD-WAN (see above). The terms used in MPLS itself are label (a short fixed-length tag), LSR (label switching router, forwards by label), LER (label edge router, adds or removes the label at the network edge), LSP (label-switched path) and FEC (forwarding equivalence class, a group of packets that are forwarded the same way and share a label). A drawback of MPLS is that it needs carrier support and a complex configuration with extra protocols such as LDP or RSVP-TE.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does an MPLS core router look at to forward a packet?

    Label switching replaces the IP lookup in the core with a simple exact-match lookup of the incoming label, which is swapped for the outgoing label.
  2. 2. What is penultimate hop popping?

    By popping the outer label one hop early, the egress router receives a packet with only the inner label (or a plain IP packet) and does less work.
  3. 3. In an MPLS Layer 3 VPN, what does the inner VPN label identify?

    The outer label moves the packet through the core, and the inner label tells the egress PE which customer routing table to use.
  4. 4. Is traffic in an MPLS VPN encrypted by default?

    MPLS provides isolation, not encryption. Customers who need secrecy over the carrier network add their own encryption.
  5. 5. What is the central idea of SD-WAN?

    SD-WAN measures every available link and steers each class of traffic to the cheapest link that meets its needs, with failover when a link degrades.

Summary

  • Leased lines are dedicated circuits: predictable, private and expensive. Full meshes need n(n-1)/2 of them, so hub-and-spoke layouts are common.
  • X.25, Frame Relay and ATM introduced shared packet networks with virtual circuits and short labels that are swapped at every hop.
  • MPLS adds a 4-byte label to IP packets. The ingress router pushes labels, core routers swap them with an exact-match lookup, and the label is popped at or before the egress. LDP and RSVP-TE distribute labels and engineer paths.
  • MPLS L3VPNs use VRFs, BGP and an inner VPN label to keep customers apart on one network. Layer 2 VPNs emulate a switch or a wire. MPLS isolates traffic, but does not encrypt it.
  • SD-WAN combines several links (MPLS, broadband, LTE) into an overlay, measures them continuously, and steers each application class according to a central policy.

Next, see how traffic can be sent to groups and given priority in Multicast, Broadcast and Quality of Service.