Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 80 ยท Network Security

IPsec, IKE, AH and ESP

Learn how IPsec secures IP traffic: IKE phases and key exchange, AH versus ESP, tunnel versus transport mode, security associations, anti-replay and safe VPN configuration.

Most internet protocols were designed without security. IP packets can be read, altered, replayed or forged as they cross routers. You can add security at the application layer (HTTPS, SSH) or at the link layer (Wi-Fi encryption), but there is also an option in the middle: protect everything at the IP layer, so every application on a host or network benefits without being modified. That is the job of IPsec (Internet Protocol Security), a suite of standards that provides authentication, integrity and optional encryption for IP traffic.

IPsec is the backbone of many corporate site-to-site VPNs and remote-access VPNs. VPNs, Wi-Fi and Remote Access Security gave a short overview; this lesson explains the moving parts: security associations, the IKE negotiation, the two protocols AH and ESP, the two modes (transport and tunnel), anti-replay protection, and how to configure IPsec safely. The Python demos model the logic without sending any network traffic. Knowing public-key cryptography and HMAC will help.

What IPsec provides

Service Meaning
Confidentiality Payload is encrypted (ESP)
Data integrity A tampered packet is detected (AH and ESP)
Data origin authentication The packet came from the claimed peer
Anti-replay A captured packet cannot be re-sent later
Access control Policy decides which traffic is protected, bypassed or dropped

Security associations

IPsec is built on security associations (SAs). An SA is a one-way agreement between two peers that records the protocol (AH or ESP), the mode, the algorithms and keys, and a lifetime. Because it is one-way, a conversation needs two (one per direction). Each SA is identified by a Security Parameter Index (SPI) carried in every packet, so the receiver knows which keys to use. Two databases drive the process:

  • Security Policy Database (SPD): rules saying which traffic must be protected, which can bypass IPsec, and which is dropped.
  • Security Association Database (SAD): the SAs currently in use.

IKE: negotiating keys

Keys and algorithms must be agreed on before protected data can flow, and the agreement itself must be authenticated. The Internet Key Exchange (IKE) protocol does this, usually on UDP port 500 (and 4500 for NAT traversal). Today you should use IKEv2, which is simpler and more robust than IKEv1.

In IKEv1 there were two explicit phases; IKEv2 keeps the same idea in a shorter exchange:

  1. Phase 1 (IKE SA): the peers negotiate algorithms, perform a Diffie-Hellman exchange to derive shared secrets, and authenticate each other with certificates, a pre-shared key, or (for remote users) EAP. The result is a secure control channel. In IKEv2 this is the IKE_SA_INIT and IKE_AUTH exchanges.
  2. Phase 2 (Child SA): over that protected channel, the peers agree on the data-plane IPsec SAs: AH or ESP, algorithms, lifetimes, and which traffic ("traffic selectors") they cover. In IKEv2 the first Child SA is created during IKE_AUTH, and CREATE_CHILD_SA makes more or rekeys them.
sequenceDiagram
    participant I as Initiator
    participant R as Responder
    I->>R: IKE_SA_INIT: proposed algorithms + DH public value + nonce
    R->>I: chosen algorithms + DH public value + nonce
    Note over I,R: Both derive shared keys (Diffie-Hellman). The channel is now encrypted.
    I->>R: IKE_AUTH (encrypted): identity + authentication + child SA proposal
    R->>I: identity + authentication + chosen child SA
    Note over I,R: Child SAs (ESP) are ready: protected traffic can flow
    I->>R: ESP packets (SPI + sequence number + encrypted data)

IKE authenticates the peers and creates the keys; ESP then carries the traffic.

Use perfect forward secrecy (PFS), which makes each Child SA use a fresh Diffie-Hellman exchange, so recording traffic and stealing a long-term key later does not reveal old sessions. Prefer certificate authentication for site-to-site and remote-access VPNs; pre-shared keys work, but a weak or reused key is a common weak point (and in some IKEv1 modes a captured exchange allows offline guessing).

AH and ESP

IPsec has two data protocols:

  • AH (Authentication Header, IP protocol 51) gives integrity and authentication for the packet, including most of the outer IP header, but no encryption.
  • ESP (Encapsulating Security Payload, IP protocol 50) gives encryption, integrity and authentication for the payload (and an optional authenticated-only mode). It does not cover the outer IP header.

In practice almost everyone uses ESP with an authenticated-encryption algorithm such as AES-GCM. AH is rarely used because ESP already authenticates, and because AH's coverage of the IP header makes it fail across NAT, which changes addresses and ports. If you need to traverse NAT, ESP is wrapped in UDP port 4500 ("NAT-T").

AH ESP
Encryption No Yes (optional, normally on)
Integrity Yes, including outer IP header Yes, payload only
Works through NAT No Yes (with NAT-T)
Common today Rare Standard

Transport mode vs tunnel mode

Transport mode Tunnel mode
What is protected The payload of the original packet; the original IP header is kept The entire original packet, which is wrapped in a new outer IP header
Typical use Host-to-host traffic (for example between two servers) Gateway-to-gateway and remote-access VPNs
Hides internal addresses No Yes: observers see only the gateway addresses
Overhead Lower Higher (extra IP header)

The next demo builds both layouts as simple data structures, to make the structure visible. Nothing here is real encryption: protect() stands in for ESP's authenticated encryption.

Python Transport vs tunnel mode, packet layouts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In tunnel mode an eavesdropper learns that the two sites talk and how much data flows, but not who inside is talking to whom, nor the ports. Traffic-volume patterns still leak, which is why sensitive systems add padding or constant-rate links.

Anti-replay protection

Every ESP and AH packet carries a sequence number. The receiver keeps a sliding window of recently seen numbers: it accepts a number only if it is new or inside the window and not yet seen, and rejects duplicates or anything too old. This stops an attacker who records packets from replaying them later (for example to repeat a payment instruction). When the sequence number nears its maximum, the peers rekey.

Python A sliding anti-replay window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that packet 4 arriving after 5 is accepted (reordering is normal), but a second 5 is a replay; after 20 arrives, the window moves forward, so 12 is now too old while 19 is still acceptable.

Choosing and auditing a configuration

IPsec has many options, and weak ones remain available for compatibility. A safe baseline today:

  • IKEv2, not IKEv1; certificates (or strong, unique, random pre-shared keys no shorter than about 20 random characters).
  • ESP with AES-GCM (128 or 256) or ChaCha20-Poly1305; avoid DES, 3DES, MD5, and SHA-1 for integrity.
  • Diffie-Hellman groups of at least 2048-bit modular groups (group 14) or, preferably, elliptic-curve groups (such as 19, 20 or Curve25519); avoid groups 1, 2 and 5.
  • PFS enabled for Child SAs; lifetimes measured in hours and volume, so keys rotate.
  • Dead-peer detection on, NAT-T on where needed.
  • Narrow traffic selectors: protect only the subnets that need it, rather than 0.0.0.0/0, and combine with firewall rules and segmentation (Firewalls, Segmentation and Zero Trust).
  • Patch gateways quickly and require MFA for remote-access users (Multi-Factor Authentication and Passkeys); VPN gateways are exposed to the internet and heavily targeted (Vulnerability Management and Patching).

A small auditor can check proposals against such a baseline:

Python Audit IKE and ESP proposals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

IPsec compared with other options

IPsec TLS-based VPN WireGuard SSH tunnel
Layer Network (IP) Application (TCP/UDP) Network (UDP) Application
Best for Site-to-site, built-in OS clients Browser or client remote access Simple, fast VPNs Admin and single-port forwarding
Complexity High (many options) Medium Low (fixed modern crypto) Low

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main purpose of IKE?

    IKE runs first, authenticates both sides, derives shared keys with Diffie-Hellman and sets up the security associations that ESP or AH then use.
  2. 2. Which protocol provides encryption?

    ESP encrypts and authenticates the payload. AH provides authentication and integrity only.
  3. 3. Why does AH usually fail across NAT?

    NAT rewrites addresses and ports, which invalidates AH's integrity check. ESP covers only the payload, and NAT-T encapsulates it in UDP.
  4. 4. A gateway-to-gateway VPN hides the internal addresses of the two sites. Which mode is used?

    Tunnel mode wraps the entire original packet in a new outer header addressed between the gateways.
  5. 5. What does the anti-replay window do?

    The receiver tracks recent sequence numbers so a recorded packet cannot be injected again.

Summary

  • IPsec protects traffic at the IP layer with authentication, integrity, optional encryption and anti-replay, using one-way security associations identified by an SPI.
  • IKE (use IKEv2) authenticates peers and negotiates keys via Diffie-Hellman; enable PFS.
  • ESP encrypts and authenticates; AH only authenticates and breaks across NAT, so ESP (with NAT-T) is the norm.
  • Transport mode protects host-to-host payloads; tunnel mode wraps the whole packet and is used for VPN gateways.
  • A sliding window of sequence numbers blocks replay.
  • Use certificates or long random keys, AES-GCM, strong DH groups, narrow selectors, and patch gateways quickly.

Next: NAT, Proxies and Network Access Control.