Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท VPNs and Tunnelling

IPsec: IKE, ESP and AH Explained

Understand IPsec: security associations, IKEv2 negotiation, ESP and AH, tunnel and transport mode, plus runnable proposal negotiation and anti-replay window demos.

IPsec is a suite of standards that secures traffic at the IP layer. Because it works below the application, it protects everything that rides on IP without changing the applications. It is the long-standing workhorse for site-to-site VPNs between offices and clouds, it is built into every major operating system, and IKEv2 with IPsec is a common option for remote-access VPNs on phones and laptops.

IPsec is not one protocol but a family, which is why it has a reputation for being complicated. This lesson breaks it into its parts: security associations, the key negotiation protocol IKE, the two data protocols ESP and AH, and the two modes. The configuration snippets are read-only examples (a Linux VPN gateway such as strongSwan); the runnable Python demos reproduce two of the key ideas: negotiating algorithms, and the anti-replay window.

The parts

flowchart TB
    A["IKE (UDP 500, and 4500 for NAT traversal)"] --> B["IKE SA: authenticated, encrypted control channel"]
    B --> C["Child SAs: one per direction"]
    C --> D["ESP packets carry the protected traffic"]

IKE sets up the keys. ESP carries the data.

Security associations (SAs)

A security association is a one-way agreement: the algorithms and keys used for traffic from A to B. A VPN therefore needs at least two child SAs (one per direction), each identified by a Security Parameter Index (SPI) carried in every packet so the receiver knows which keys to use. The IKE SA is the separate control channel that negotiated them.

IKE: negotiating and authenticating

The Internet Key Exchange (IKEv2 is the current version; IKEv1 is legacy) runs over UDP port 500 (4500 when NAT is in the path) and does four jobs:

  1. Negotiates algorithms (a proposal list: encryption, integrity, Diffie-Hellman group).
  2. Authenticates both peers with a pre-shared key (fine for small setups, if long and random), certificates (better at scale), or EAP for remote users.
  3. Derives fresh session keys using Diffie-Hellman, so that recorded traffic cannot be decrypted later if a long-term key leaks (perfect forward secrecy).
  4. Rekeys periodically, replacing keys before they have protected too much data.

IKEv2 does this in two request/response pairs: IKE_SA_INIT (negotiate, exchange DH values, nonces) and IKE_AUTH (prove identities, create the first child SA). Later CREATE_CHILD_SA exchanges rekey or add child SAs.

ESP: the data protocol

Encapsulating Security Payload provides confidentiality, integrity, authentication of origin and replay protection. Modern ESP uses an AEAD cipher such as AES-GCM or ChaCha20-Poly1305, which encrypts and authenticates in one step. An ESP packet has an SPI, a sequence number, the encrypted payload, padding and an integrity tag. In tunnel mode, the payload is the whole original IP packet.

AH: authentication only

The Authentication Header authenticates the packet (including parts of the outer IP header) but does not encrypt. It breaks under NAT, because NAT changes the addresses it authenticated, and ESP with integrity gives what most designs need. AH is rarely used today; you should recognise it but you will almost always choose ESP.

Modes

  • Tunnel mode protects the whole original packet and adds a new IP header; used for gateways and VPNs.
  • Transport mode protects only the payload; used between two hosts directly (and, for example, with L2TP/IPsec).

What good IPsec settings look like

A modern, defensible baseline:

  • IKEv2, not IKEv1 (avoid aggressive mode entirely).
  • AES-256-GCM or ChaCha20-Poly1305 for ESP; SHA-256 or better for anything that still needs a separate integrity hash.
  • Diffie-Hellman groups 14 (2048-bit) or higher, or elliptic-curve groups such as 19/20/21 or Curve25519 (group 31). Avoid groups 1, 2 and 5.
  • Perfect forward secrecy enabled for child SAs.
  • Certificates or strong, unique, long pre-shared keys; never a short shared password.
  • Reasonable lifetimes, for example hours for the IKE SA and shorter for child SAs.
  • Dead peer detection so the tunnel recovers when the other side disappears.
TEXT
# strongSwan swanctl.conf (read-only example, documentation addresses)
connections {
  branch {
    version = 2
    local_addrs  = 203.0.113.1
    remote_addrs = 198.51.100.2
    proposals = aes256gcm16-prfsha384-ecp384
    local  { auth = pubkey; certs = gateway-a.pem }
    remote { auth = pubkey; id = "gateway-b.example.net" }
    children {
      lan {
        local_ts  = 10.0.0.0/24
        remote_ts = 10.1.0.0/24
        esp_proposals = aes256gcm16-ecp384
        mode = tunnel
        start_action = start
        dpd_action = restart
      }
    }
  }
}

The local_ts and remote_ts lines are traffic selectors: they decide which packets enter the tunnel, a security policy that behaves like a firewall rule. Mismatched selectors are the most common reason tunnels come up but pass no traffic.

Demo 1: negotiating a proposal

IKE does not pick algorithms by magic: the initiator sends an ordered list of proposals and the responder picks the first one it also supports (or, in some implementations, follows its own order). The program below negotiates and shows why weak entries in your proposal list are dangerous: if the peer supports them, an attacker who can tamper with the exchange may push you down to the weak choice, so remove them from both sides.

Python Negotiate an IPsec proposal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Failing closed is good: a tunnel that cannot agree on strong algorithms should not come up at all. The last row shows it: the gateway no longer offers any legacy proposal, so the old client gets no tunnel until it is upgraded. The second row shows a quiet downgrade: the peers settle on the best proposal they share, which is weaker than the modern ones, so review which fallbacks you really need and remove the weak entries from both lists.

Demo 2: the anti-replay window

An attacker who records ESP packets could replay them later, perhaps repeating a command. ESP prevents this with a sequence number in every packet and a sliding window on the receiver: packets older than the window are rejected, and packets inside the window are accepted only once.

Python A sliding anti-replay window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Out-of-order packets inside the window (like 4 after 5) are fine, since networks reorder traffic; repeats (the second 3 and the second 20) and packets far behind the window (12 after 20 moved the window forward) are rejected. Because the sequence number is part of what is authenticated, an attacker cannot simply rewrite it. Real ESP uses a 32- or 64-bit number (with an extended 64-bit option) and a window of at least 32 packets, and it triggers a rekey before the counter would wrap.

NAT traversal and other practicalities

  • ESP is IP protocol 50, not TCP or UDP, so plain NAT devices cannot translate it. NAT-T detects NAT during IKE and wraps ESP inside UDP port 4500.
  • Firewalls in front of a gateway must allow UDP 500, UDP 4500 and (if no NAT-T) protocol 50.
  • MTU: ESP adds about 50 to 70 bytes. Set the tunnel MTU (and clamp TCP MSS) to avoid stalled large transfers.
  • Policy-based versus route-based VPNs: policy-based tunnels match traffic selectors from firewall policy; route-based tunnels create a virtual interface (VTI) and use normal routing, which is more flexible for dynamic routing and multiple subnets.
  • Troubleshooting: check that both ends agree on IDs, proposals and selectors; the logs of the responder usually say which one differs (NO_PROPOSAL_CHOSEN, TS_UNACCEPTABLE, AUTHENTICATION_FAILED).

A pre-shared key reused across many tunnels, or one that is short and guessable, is a classic weakness: a captured IKEv1 aggressive-mode handshake can be attacked offline. Prefer IKEv2 with certificates, or long random keys unique to each tunnel.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a security association (SA) in IPsec?

    An SA is a one-directional set of parameters. A bidirectional VPN needs two child SAs, plus the IKE SA that negotiated them.
  2. 2. Which protocol should you normally choose for protecting data in modern IPsec VPNs?

    ESP provides confidentiality, integrity and replay protection. AH offers no encryption and does not work with NAT.
  3. 3. What does perfect forward secrecy give you?

    PFS derives new session keys from ephemeral exchanges, so compromising the long-term key does not reveal past sessions.
  4. 4. In the anti-replay demo, why is a packet far behind the window rejected?

    The receiver only remembers a bounded window of sequence numbers. Anything older is dropped to avoid accepting replays.
  5. 5. IPsec tunnel comes up but no traffic passes. What is a common cause?

    Traffic selectors decide what enters the tunnel. If the peers disagree, the child SA may not be created or no packets match it.

Summary

  • IPsec secures traffic at the IP layer; IKE negotiates and authenticates, ESP carries protected data, AH is rarely used.
  • Security associations are one-way; SPIs select the keys; tunnel mode wraps whole packets.
  • Use IKEv2, AEAD ciphers, strong DH groups with PFS, certificates or long unique PSKs, sensible lifetimes and DPD.
  • Proposal negotiation should fail closed rather than fall back to weak algorithms; anti-replay windows reject repeated and stale packets.
  • Plan for NAT-T, MTU, traffic selectors and policy- versus route-based designs.

Next: WireGuard and OpenVPN