Skip to content
Tutorials Center
My learning

Topics

Lesson 72 of 79 ยท How the Web Works and Real-World Projects

Project: Ethernet, IP and TCP Packet Parser

Build a packet parser in Python: decode raw hex bytes layer by layer with struct, read TCP options and verify the IPv4 and TCP checksums.

Tools like Wireshark and tcpdump look magical until you realize what they do: they take a long run of bytes, peel off one header at a time, and print the fields in plain language. In this project you will write that peeling logic yourself. Your parser will turn a hex dump of an Ethernet frame into a readable summary: MAC addresses, IP addresses, TTL, TCP ports, flags, options and payload. It will also verify the checksums, so you can see exactly which layer notices which kind of damage.

The project ties together everything from the earlier chapters: Ethernet frames, the IPv4 header and the TCP segment. The only new tool is Python's struct module, which converts between bytes and numbers. Everything runs live in your browser.

The plan

A captured frame is a single byte string. Each layer has a header whose fields tell you how long it is and what comes next:

flowchart LR
    F["Frame bytes"] --> E["Ethernet header<br/>14 bytes<br/>type 0x0800?"]
    E --> I["IPv4 header<br/>IHL x 4 bytes<br/>protocol 6?"]
    I --> T["TCP header<br/>offset x 4 bytes"]
    T --> P["Payload"]

Each header says what the next one is: the Ethernet type field selects IPv4, the IP protocol field selects TCP, and the lengths say where each header ends.

The parser will be a chain of small functions, each taking bytes and returning (fields, remaining_bytes). That shape makes it easy to test each layer on its own, and easy to add layers later (UDP, ICMP, IPv6).

Reading bytes with struct

A format string describes the layout of a header. Network protocols use big-endian ("network byte order"), written ! in struct. The letters are sizes: B is 1 byte, H is 2 bytes, I is 4 bytes, and 6s means 6 raw bytes. The 14-byte Ethernet header is therefore "!6s6sH": destination MAC, source MAC and a 2-byte type.

Start by printing a hex dump (the way tools show packets) and decoding the Ethernet header by hand.

Python Step 1: a hex dump and the Ethernet header

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The text column on the right shows why hex dumps are so useful for debugging: the payload is plain ASCII (GET / HTTP/1.1), while the headers look like noise. The first byte after the Ethernet header is 45: version 4 and an IP header length of 5 words (20 bytes).

A frame captured on the wire also ends with a 4-byte frame check sequence (FCS), but the operating system normally strips it before tools see the frame, so your hex dump has no CRC. Frames shorter than 60 bytes are padded with zeros by the sender or the network card; you will see that padding in one of the examples below.

The complete parser

Now write one function per layer.

Ethernet gives the destination and source MAC and the EtherType. 0x0800 means IPv4 follows and 0x0806 means ARP.

IPv4 is the layer with the most traps:

  • The first byte holds two 4-bit values. ver_ihl >> 4 is the version and (ver_ihl & 0x0F) * 4 is the header length in bytes, because IHL counts 32-bit words. Options can make the header longer than 20 bytes.
  • The total length field says where the IP packet ends. Anything after it in the frame is Ethernet padding and must be ignored. Slicing with data[ihl:total] handles this.
  • The 16-bit flags-and-offset field is split: 0x4000 is the Don't Fragment bit, 0x2000 is More Fragments, and the low 13 bits are the fragment offset in units of 8 bytes.
  • To verify the header checksum, you compute the Internet checksum over the whole header including the checksum field. A correct header gives 0.

TCP has the data offset (header length in words, in the top 4 bits of byte 12), a flags byte, and an options area between byte 20 and the offset. Options use a kind-length-value layout, with one-byte special cases: kind 0 is end-of-list and kind 1 is a NOP used for padding. The TCP checksum covers the segment plus a pseudo-header containing the source address, destination address, protocol number and segment length. This is why a NAT that rewrites an IP address must also fix the TCP checksum.

The Internet checksum is the same for both layers: add all 16-bit words, fold the carries back into the low 16 bits, and invert the result.

The parser below also refuses to guess: when data is too short it raises a ParseError instead of reading garbage. A real packet parser must be defensive, because captured data is untrusted input and attackers send malformed packets on purpose.

Python Step 2: a layered packet parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the five results.

  • Frame 1 is a PSH+ACK segment carrying an HTTP request. Its sequence number is 5000 and its acknowledgment number says the next server byte it expects is number 1000.
  • Frame 2 is the very first segment of a connection: only SYN is set. The options decode to an MSS of 1460, a window scale of 7 and SACK permission, with NOPs padding the option block to a multiple of four bytes. These are the options you met in the handshake lesson.
  • Frame 3 is a bare ACK of 60 bytes on the wire, but the IP total length says only 40 bytes. The six extra zero bytes are Ethernet padding, and the parser correctly ignored them: no payload is printed.
  • Frame 4 is an ARP request. The parser stops at the Ethernet layer because it does not know ARP. A natural extension is to decode it.
  • Frame 5 is a capture cut short (the "snaplen" of a capture tool can truncate frames). The parser reports exactly which layer ran out of bytes instead of crashing.

Which layer notices which damage?

Checksums exist so that a receiver can tell that something went wrong. They are not equally powerful, and they protect different things. This experiment builds on the parser's checksum function and corrupts the HTTP frame in several ways. A short check function looks at the IPv4 header checksum and at the TCP checksum (using the pseudo-header).

Python Step 3: breaking the frame on purpose

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each line teaches something.

  1. A flipped payload bit leaves the IP header untouched, so only the TCP checksum fails. The IP checksum covers the header only, never the payload.
  2. Routers lower the TTL of every packet, so they must recompute the IP checksum at every hop. A wrong TTL without a fix breaks the IP checksum but not TCP, because TTL is not part of the TCP pseudo-header.
  3. Rewriting an IP address breaks both checksums, since the address is in the IP header and in the TCP pseudo-header. A NAT device updates both. Fixing only the IP checksum leaves TCP broken, which is a classic NAT bug.
  4. Swapping two 16-bit words goes undetected. Addition is commutative, so the Internet checksum cannot see reordered words. It is a weak, cheap check, designed for speed on 1980s hardware. That is why the Ethernet CRC (strong against burst errors) and, above all, TLS (cryptographic integrity) exist as additional layers.

Ideas to extend it

  • Add UDP and ICMP. UDP has only a 8-byte header; use protocol number 17 and the same pseudo-header idea. ICMP is protocol 1.
  • Decode ARP. The 28-byte ARP body has hardware type, protocol type, operation and the sender/target addresses.
  • Read a real pcap file. A capture file starts with a 24-byte global header followed by records, each with a 16-byte record header (timestamps, captured length) and then the frame. You can loop over them with the same struct approach. The packet capture lesson shows what Wireshark does with those records.
  • Support IP options and fragments. Reassemble fragments with the same identification value, using the offsets you already decode.
  • Follow a stream. Group segments by their 4-tuple (source and destination address and port), sort by sequence number and print the reassembled payload, which is what "Follow TCP Stream" does.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the struct format string "!6s6sH" describe?

    The exclamation mark selects network (big-endian) byte order, 6s reads six raw bytes (a MAC address) twice, and H reads a 2-byte unsigned integer (the EtherType).
  2. 2. An IPv4 header starts with the byte 0x46. What is the header length?

    The low nibble is the IHL, 6 words of 4 bytes each, which is 24 bytes (20 bytes plus 4 bytes of options). The high nibble, 4, is the version.
  3. 3. A frame has 60 bytes on the wire but the IP total length field says 40. What are the last 6 bytes?

    Ethernet requires at least 60 bytes (before the CRC), so short frames are padded. The IP total length tells the parser where the real packet ends.
  4. 4. Why does the TCP checksum include a pseudo-header with the IP addresses?

    The pseudo-header ties the segment to the addresses and protocol it was meant for. As a result, a device that changes an address (such as a NAT) must also update the TCP checksum.
  5. 5. Which damage does the Internet checksum fail to detect?

    The checksum is a sum of 16-bit words, and addition does not depend on order, so reordering words leaves it unchanged. Stronger protection comes from the Ethernet CRC and from TLS.

Summary

  • Parsing a packet means peeling headers: Ethernet's type selects IPv4, IP's protocol field selects TCP, and length fields (IHL, data offset, total length) say where each header ends.
  • struct.unpack with the ! prefix reads network byte order; bit tricks (>> 4, & 0x0F, masks) pull out nibble and flag fields.
  • Always bounds-check. Captured data can be truncated, padded or deliberately malformed, so a parser should report errors instead of crashing or guessing.
  • The IP checksum covers the header only and is recomputed at every hop; the TCP checksum covers the segment plus a pseudo-header, so NATs must fix both.
  • The Internet checksum is cheap but weak (it misses swapped words), which is why stronger checks exist at other layers.

Now switch from bytes to live connections with a TCP chat server and client.