Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 79 ยท Routing

BGP and Inter-Domain Routing

Understand how BGP connects the Internet's autonomous systems: eBGP and iBGP, path attributes, routing policy, route leaks, hijacks and anycast, with live Python.

OSPF and RIP work inside one organization. The Internet, however, is thousands of independently run networks, and they must agree on how to reach each other without sharing their internal maps or trusting each other's cost numbers. The protocol that glues them together is the Border Gateway Protocol (BGP), currently version 4. Every time you open a web page hosted by another company, BGP routes decided which networks your packets cross.

BGP is different from the protocols in the previous two lessons. It does not look for the "shortest" path. It looks for the path that fits each network's business and policy decisions: who it pays, who it peers with and what traffic it wants to carry. This lesson explains the pieces and shows what happens when the trust that BGP depends on fails.

Autonomous systems

An autonomous system (AS) is a set of networks and routers under one administrative control that presents one routing policy to the outside world. Examples are an ISP, a university, a cloud provider or a large company. Each AS has an AS number (ASN), assigned by the regional Internet registries. Originally these were 16-bit numbers, and today they are 32-bit. Numbers 64496-64511 are reserved for documentation, and that is what the examples in this lesson use. Numbers 64512-65534 are private, for internal use.

Networks inside an AS run an interior gateway protocol (IGP) such as OSPF to find the best internal paths. Between ASes, they run BGP, the exterior gateway protocol. The business relationships between ASes decide what BGP is allowed to carry:

Relationship Who pays Typical routes exchanged
Customer to provider (transit) The customer pays the provider The provider gives the customer all routes; the customer announces only its own and its customers' prefixes
Peer to peer Usually nobody (settlement-free) Each side announces its own and its customers' prefixes, not its providers'

A network will happily carry traffic that earns money (from a customer) and will avoid carrying traffic that costs money (between two of its providers). This is the reason why paths on the Internet are not always the geographically shortest. See the structure of the Internet for the bigger picture of tiers and exchange points.

flowchart TB
    T["Transit provider<br/>AS64501"] --- X["Your ISP<br/>AS64502"]
    T --- Y["Content network<br/>AS64520"]
    X --- C["Your home<br/>(customer)"]
    X -. "peering" .- Y

Your ISP reaches the content network over a settlement-free peering link if one exists, or buys transit through the provider above.

eBGP and iBGP

BGP runs over a TCP connection on port 179 between two routers, which are called peers or neighbors. Using TCP means BGP does not need its own reliability layer: the routers exchange full tables once and then only send updates (new routes and withdrawals), with periodic keepalives to prove the session is alive.

  • eBGP (external BGP) runs between routers in different ASes, usually over a directly connected link. When a route crosses an eBGP session, the sending router prepends its own ASN to the AS path.
  • iBGP (internal BGP) runs between routers of the same AS to spread the routes learned at the borders to the interior routers. iBGP does not change the AS path. To avoid loops inside the AS, a router does not re-advertise a route learned from one iBGP peer to another iBGP peer, so iBGP normally requires a full mesh of sessions, or a route reflector to reduce it.

Path attributes and route selection

A BGP route is a prefix (such as 203.0.113.0/24) plus path attributes that describe it:

  • AS_PATH: the list of ASNs the announcement passed through, newest first. It is used to detect loops and as a rough distance measure.
  • NEXT_HOP: the IP address of the router to forward packets to.
  • LOCAL_PREF: a number that an AS sets for its own routers to express a preference. It is shared inside the AS only. Higher wins.
  • MED (multi-exit discriminator): a hint from a neighbor AS about which of several links into it you should prefer. Lower wins.
  • ORIGIN: how the route entered BGP.
  • COMMUNITY: tags that carry policy instructions, for example "do not announce this to peers".

When a router hears several routes to the same prefix, it picks one using an ordered list of tie-breakers. A simplified version of the usual order is:

  1. Highest LOCAL_PREF (this is where business policy lives).
  2. Shortest AS_PATH.
  3. Lowest ORIGIN type.
  4. Lowest MED (between routes from the same neighboring AS).
  5. Prefer eBGP routes over iBGP routes.
  6. Lowest IGP cost to the next hop (nearest exit).
  7. Lowest router ID, as the last tie-breaker.

Some vendors add extra steps, for example a "weight" that is local to one router. The simulation below applies the first four rules. It also applies the BGP loop rule: a router rejects any route whose AS path already contains its own ASN, because the announcement has already gone around a circle. This is also why BGP, being a path-vector protocol, avoids the count-to-infinity problem of distance-vector routing.

Python Choosing the best BGP route

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The peer route wins at first even though its AS path is the longest, because LOCAL_PREF is checked before the path length. Policy beats distance. The "Loop" route would have won with LOCAL_PREF 300, but it is thrown out before the comparison starts.

Policy: what to announce and accept

Operators control BGP with filters and attribute changes on every session. The rules of thumb for a normal network are:

  • Announce your own prefixes to everyone you want to be reachable from.
  • Announce customer prefixes to providers and peers.
  • Never announce one provider's or peer's routes to another provider or peer, because that would turn you into free transit for them.
  • Give customer routes a higher LOCAL_PREF than peer routes, and peer routes a higher LOCAL_PREF than provider routes. Money-making paths come first.

AS-path prepending is a common trick for influencing incoming traffic: you add your own ASN to the path several times, which makes the route look longer so other networks prefer your other links.

Route leaks and hijacks

BGP was designed in a smaller, friendlier Internet. By default, a router believes what its neighbor announces. Two kinds of failure follow from that.

  • A route leak happens when a network announces routes it should not, usually through a misconfigured filter, for example a small customer passing one provider's routes to another provider. Traffic suddenly flows through a small network that cannot carry it, and large parts of the Internet slow down or disappear.
  • A prefix hijack happens when a network announces prefixes it does not own, by mistake or on purpose. Traffic can be redirected to the wrong place, dropped or inspected.

The strongest trick is the more-specific announcement. Forwarding uses longest-prefix match (see CIDR and route aggregation), so a hijacker who announces a smaller prefix than the owner wins, even with a longer AS path. The example shows this, then applies a simple origin check.

Python A more-specific prefix hijack

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Half of the owner's address block now goes to the hijacker, while the other half is still correct, which makes the problem hard to notice. The check at the end is the idea behind RPKI route origin validation (ROV): address owners publish signed ROAs (route origin authorizations) that say which ASN may originate a prefix and up to which length, and routers drop announcements that contradict them. Other defenses are strict prefix filters for customers, maximum-prefix limits on sessions, and monitoring services that alert owners when someone else announces their prefix. Full path validation (BGPsec) exists but is rarely deployed.

BGP incidents make the news regularly. They are almost always one of the two failures above: a filter was missing, and a neighbor's announcement was trusted. The protocol is not broken; it simply has no built-in way to check whether an announcement is true.

Anycast

Anycast means that many servers in different places share the same IP address, and every one of them announces that prefix with BGP. Each router on the Internet picks the "closest" announcement (usually the shortest AS path or the nearest exit), so users automatically reach a nearby server. Anycast is how the DNS root servers and public resolvers answer from hundreds of sites, and how CDNs and DDoS protection services spread an attack across many locations. If one site fails, it withdraws its announcement and traffic flows to the next-closest one.

Anycast works best for short, stateless exchanges such as DNS queries. A long TCP connection can break if routing changes mid-stream and moves the user to another site, so operators using anycast for web traffic add care and extra mechanisms. You will see it again with DNS and CDNs.

Multihoming and route dampening

A network that connects to two or more providers is multihomed. It gets resilience (if one provider fails, the other carries on), more bandwidth, and the ability to prefer cheaper or faster links. With BGP the customer announces its prefixes to both providers (in its own AS, or with a private AS number if it has none), and it influences the incoming path with tools such as AS path prepending, while using local preference to choose the outgoing path. The drawbacks are more complex configuration, the cost of two connections, and the risk of becoming an accidental transit AS between the two providers unless export filters are set, which is how route leaks start.

Route flap dampening protects routers from unstable routes. Each time a prefix is withdrawn and announced again (flaps), the router adds a penalty to it. When the penalty passes the suppress threshold, the route is hidden and not advertised to neighbors. The penalty then decays exponentially with a configured half-life, and the route returns when it falls below the reuse threshold. This stops one unstable link from triggering updates across the whole Internet, but it can also delay a good route that flapped a few times by accident, so modern practice uses it with relaxed settings or not at all.

Python Route flap dampening penalty over time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three quick flaps push the penalty over 2000 and the route is hidden. It becomes usable again only after the penalty has decayed below 750, which takes tens of minutes.

Route redistribution and policy-based routing

A network may run several routing protocols, for example OSPF in the campus and BGP toward the provider. Route redistribution copies routes learned by one protocol into another so that both sides can reach each other. Each protocol has its own metric, so the router must assign a seed metric to redistributed routes. Problems it can introduce are routing loops (routes redistributed in both directions at two points can circle back), suboptimal paths because metrics are not comparable, and loss of information. Mitigation: redistribute at one place, or filter with route maps and tags, so a route that has been redistributed is not redistributed again.

Normal routing looks only at the destination address. Policy-based routing (PBR) chooses the next hop using other fields, such as the source address, the protocol or the port. An office could send guest Wi-Fi traffic through a cheap link and voice traffic through the low-latency one. It is flexible but harder to troubleshoot, because the path no longer follows from the routing table alone, and a misconfigured policy can create loops or black holes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does BGP optimize for, compared with OSPF?

    BGP carries policy: attributes like LOCAL_PREF let each AS prefer customers over peers over providers. Shortest AS path is only one tie-breaker, after policy.
  2. 2. A router receives a BGP route whose AS path contains its own ASN. What does it do?

    Seeing your own ASN in the AS path means the route has already passed through you. Rejecting it is BGP's loop prevention.
  3. 3. Two routes have equal LOCAL_PREF. One has an AS path of 2 ASNs, the other of 4. Which wins?

    After LOCAL_PREF, the shortest AS_PATH is the next rule in the selection process.
  4. 4. Why can announcing a more-specific prefix hijack traffic even when the real owner announces the larger block?

    Packets are forwarded using the longest matching prefix, regardless of AS-path length. A /24 beats a /23 for the addresses it covers.
  5. 5. Which is the correct role of iBGP?

    iBGP runs between routers of one AS so that interior routers know the externally learned routes. It does not add the ASN to the path, and an IGP still finds the paths to the next hops.

Summary

  • An autonomous system is a network under one administration, identified by an ASN; BGP routes between ASes, and an IGP such as OSPF routes inside them.
  • BGP runs over TCP port 179; eBGP is between ASes and prepends the ASN, iBGP spreads routes inside an AS.
  • Routes carry attributes (AS_PATH, NEXT_HOP, LOCAL_PREF, MED, communities); selection checks LOCAL_PREF first, then AS path length and other tie-breakers.
  • Policy follows money: prefer customers, then peers, then providers, and never leak one provider's routes to another.
  • Route leaks and prefix hijacks happen because announcements are trusted; filters, RPKI origin validation and monitoring reduce the risk.
  • Anycast announces one prefix from many places so users reach a nearby site.

Next, learn how one packet can reach many receivers in Multicast, Broadcast and Quality of Service.