Skip to content
Tutorials Center
My learning

Topics

Lesson 65 of 79 ยท Network Security

Firewalls, VPNs and Intrusion Detection

Learn how packet filters and stateful firewalls decide what to allow, how DMZs, VPN tunnels (IPsec, WireGuard) and IDS/IPS work, and try a live firewall simulator.

Encryption protects data in transit, but it does not decide who may talk to whom. That is the job of a firewall: a checkpoint that looks at traffic crossing a boundary and applies rules to it. A VPN does a different but related job. It builds an encrypted tunnel across an untrusted network, so that two sites or a remote worker behave as if they were on the same private network. And an intrusion detection system watches what gets through, looking for behavior that should not be there.

This lesson builds on security fundamentals (defense in depth) and cryptography for networking. By the end you will be able to read a firewall rule list, explain why stateful inspection matters, describe how a DMZ is laid out, compare IPsec and WireGuard, and tell an IDS from an IPS.

Packet filters

The simplest firewall is a packet filter. It examines each packet on its own and compares the header fields against an ordered list of rules. The usual fields are:

  • source and destination IP address,
  • protocol (TCP, UDP, ICMP),
  • source and destination port,
  • direction (inbound or outbound) and the interface,
  • for TCP, some flag bits such as SYN.

Rules are checked from top to bottom and the first match wins. Every list ends with a default policy: either default deny (drop anything not explicitly allowed) or default allow. Default deny is the safe choice, because a forgotten rule then blocks something instead of exposing it.

TEXT
# order  action  proto  source           destination      dport
1        allow   tcp    any              203.0.113.10     443
2        allow   tcp    10.0.0.0/8       203.0.113.10     22
3        deny    any    any              any              any    (default)

Rule 1 lets anyone reach the public web server over HTTPS. Rule 2 lets only the internal network use SSH. Everything else dies at rule 3. Order matters: if rule 3 were placed first, nothing would ever be allowed.

A stateless filter has a weakness. A reply to an outbound request arrives from a port the filter never opened. To allow web browsing, a stateless filter would have to accept any incoming TCP packet with a high destination port, which also lets attackers in.

Stateful inspection

A stateful firewall remembers connections. When an inside host opens a TCP connection to the outside, the firewall records the 5-tuple (protocol, source IP, source port, destination IP, destination port) in a connection table. Packets that belong to a known connection, including the replies in the opposite direction, are allowed automatically. Packets that claim to be replies but match no entry are dropped. Entries disappear when the connection closes (see TCP teardown) or after an idle timeout.

This gives the policy people actually want: "inside machines may start connections to the Internet, but the Internet may not start connections to inside machines". UDP and ICMP have no handshake, so the firewall simply tracks "recently seen" flows with short timeouts. The table works much like the one in NAT, and in practice the same device usually does both jobs.

Python A first-match firewall with a connection table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Application-layer and next-generation firewalls

Ports are only a hint about what a program is really doing. An application firewall (or next-generation firewall) reads further into the packet: it can recognize a protocol regardless of port, filter by URL or domain, check a user identity, or decrypt TLS to scan its contents. A web application firewall (WAF) sits in front of a website and blocks requests that look like SQL injection or cross-site scripting. The price is more processing per packet and, for TLS inspection, a trust trade-off, because the firewall must act as a deliberate man in the middle.

Firewalls also run on single machines. A host firewall (Windows Defender Firewall, nftables, ufw) protects one computer even when it sits on a hostile network, which is a form of defense in depth.

The DMZ

Public servers must accept connections from the Internet, but if one is compromised you do not want the attacker to land on your internal network. The classic answer is a DMZ (demilitarized zone): a separate network segment for exposed services, with firewall rules between all zones.

flowchart LR
    NET["Internet"] --> FW{"Firewall"}
    FW --> DMZ["DMZ<br/>web, mail, DNS"]
    FW --> LAN["Internal LAN<br/>staff, databases"]
    DMZ -. "limited: app to DB only" .-> LAN

Internet traffic may reach the DMZ; the DMZ may reach only a narrow set of internal services; the internal network is never reachable directly from outside.

Typical rules: the Internet may connect to the DMZ on ports 80 and 443; the DMZ web server may connect to the database port inside; the internal network may connect outward; the DMZ may not start connections into the LAN otherwise. If the web server is hacked, the attacker is still boxed in.

VPNs and tunneling

A tunnel carries one packet inside another. The original packet becomes the payload of a new outer packet addressed between the two tunnel endpoints. Add encryption and authentication and you have a virtual private network (VPN). Two common uses:

  • Site-to-site VPN: two offices connect their networks through a pair of gateways over the Internet.
  • Remote access VPN: a laptop opens a tunnel to the company gateway and receives an internal address.
flowchart LR
    A["Laptop<br/>10.8.0.2 inner"] --> E1["VPN client<br/>encrypt + wrap"]
    E1 -->|"outer IP: client to gateway<br/>payload encrypted"| E2["VPN gateway<br/>unwrap + decrypt"]
    E2 --> S["Internal server<br/>10.0.0.5"]

The inner packet (10.8.0.2 to 10.0.0.5) is hidden inside an encrypted outer packet that crosses the Internet.

IPsec

IPsec is a suite that secures traffic at the IP layer, so applications need no changes. Its main parts:

  • IKE (Internet Key Exchange) authenticates the peers, using certificates or pre-shared keys, and negotiates keys and algorithms. The negotiated agreement for one direction is a security association (SA).
  • ESP (Encapsulating Security Payload) encrypts and authenticates packets. AH only authenticates and is rarely used.
  • Transport mode protects only the payload of the original packet (host to host). Tunnel mode wraps the whole original packet in a new IP packet and is what gateways use.

IPsec is standard and widely supported, and also large and complex, with many options to configure correctly.

WireGuard

WireGuard is a newer VPN designed to be small. It uses a fixed, modern set of primitives (Curve25519 for key exchange, ChaCha20-Poly1305 for encryption, BLAKE2s for hashing) rather than negotiating algorithms. Each peer has a key pair and a list of allowed IPs: the addresses it may send from, which also serve as the routing table for the tunnel. It runs over UDP, stays silent to unauthenticated packets, and its codebase is a small fraction of the size of IPsec stacks, which makes it easier to audit.

TEXT
# a typical WireGuard configuration (read-only)
[Interface]
PrivateKey = <this machine's secret key>
Address    = 10.8.0.2/32

[Peer]
PublicKey  = <the gateway's public key>
Endpoint   = vpn.example.com:51820
AllowedIPs = 10.0.0.0/16

Other VPN styles you will meet: TLS-based VPNs (OpenVPN, SSL VPN portals), which ride on TLS, and application-level tunnels such as SSH port forwarding.

What a VPN is used for, and the protocols you can choose

People use VPNs to protect traffic on untrusted networks such as public Wi-Fi, to reach a company network from home, to join sites together, to hide their location from the sites they visit (which can bypass geographic restrictions and can reduce throttling by an ISP, since the ISP only sees an encrypted tunnel), and to hide their activity from the local network. Two caveats: a VPN only moves trust from your local network to the VPN operator, who can see where your traffic goes, and some services and countries block VPN endpoints.

How it works in four steps: the client connects and authenticates to the VPN server, the two agree on keys, the client encapsulates its packets (tunneling) inside encrypted packets addressed to the server, and the server decrypts them and forwards them to the real destination, then does the reverse for replies.

Protocol Strengths and weaknesses
PPTP Old and built into early Windows. Fast but insecure; do not use
L2TP/IPsec Secure when configured well, widely supported, but double encapsulation makes it slower and UDP 500/4500 can be blocked
IKEv2/IPsec Fast and good at reconnecting when the device changes networks (phones moving between Wi-Fi and mobile data)
OpenVPN Open source, flexible, works over TCP 443 to pass strict firewalls, a little heavier
SSTP Microsoft's tunnel inside TLS on TCP 443, passes most firewalls, mainly for Windows
WireGuard Small code base, modern cryptography, very fast, simple configuration

The terms voluntary and compulsory tunneling distinguish who starts the tunnel. In voluntary tunneling the user's own computer runs the VPN client and builds the tunnel to the server, as with a laptop. In compulsory tunneling a network device, such as the ISP's access server, creates the tunnel for the user without the user's action, which gives the provider control but gives the user no choice about where the traffic goes. PPTP and L2TP supported both styles.

Intrusion detection and prevention

A firewall decides on headers. An intrusion detection system (IDS) inspects traffic or host logs for signs of attack and raises alerts. An intrusion prevention system (IPS) sits inline and can also block.

Approach How it works Strength Weakness
Signature-based Matches traffic against patterns of known attacks Few false alarms for known threats Misses new attacks
Anomaly-based Learns a baseline of normal behavior, flags deviations Can spot new attacks More false positives

Network IDS (NIDS) watch a network tap or mirror port; host IDS (HIDS) watch one machine's files and logs. Encrypted traffic limits what a NIDS can see, so much detection has moved to endpoints and logs. An IPS that is inline must be tuned carefully, because a false positive blocks legitimate users.

Zero trust

Traditional designs trusted anything inside the perimeter. That fails when an attacker gets one laptop, or when staff work from anywhere. Zero trust drops the idea of a trusted network: every request is authenticated and authorized based on identity, device health and context, access is limited to what is needed (least privilege), and traffic is encrypted even inside the office. In practice it means strong identity checks, per-application access instead of whole-network VPN access, and micro-segmentation of internal networks.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A firewall rule list is checked top to bottom. What happens at the first rule that matches a packet?

    First match wins, which is why rule order matters and why a broad rule placed too early can shadow more specific ones.
  2. 2. What advantage does a stateful firewall have over a stateless packet filter?

    By recording each connection in a table, it recognizes legitimate replies and drops packets that merely pretend to be replies.
  3. 3. Why are public-facing servers usually placed in a DMZ?

    The DMZ is a separate segment with strict rules in both directions, which limits the damage of a breach.
  4. 4. Which statement about IPsec tunnel mode is correct?

    Tunnel mode encapsulates the complete original packet, which is why gateways use it to join networks.
  5. 5. How does an IPS differ from an IDS?

    Detection raises alerts; prevention is placed in the traffic path and can drop packets, with the risk of blocking legitimate traffic after a false positive.

Summary

  • A packet filter compares headers with an ordered rule list; the first match wins and a default-deny policy ends the list.
  • Stateful firewalls track connections so replies are allowed automatically while unsolicited inbound packets are dropped. Application firewalls and WAFs look deeper than ports.
  • A DMZ isolates exposed servers from the internal network with firewall rules between zones.
  • VPNs encapsulate and encrypt packets in a tunnel. IPsec (IKE, ESP, tunnel and transport modes) is the standard suite; WireGuard is a small, modern alternative with fixed algorithms and allowed-IP lists.
  • An IDS detects (by signatures or anomalies) and an IPS blocks inline. Zero trust replaces perimeter trust with per-request authentication and least privilege.

Next, see what attackers actually try against these defenses in common attacks and how to defend.