Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 40 ยท Networking and Streams

Framing Messages over a Stream

Learn how to split a TCP byte stream into messages in asyncio: newline framing, length-prefixed binary frames, delimiters, size limits and incomplete reads.

TCP delivers a stream of bytes, not a stream of messages. If a client sends "hello" and then "world", the server may see b"helloworld" in one read, or b"hel" and b"loworld" in two. Any protocol you build on streams must therefore define framing: a rule that says where one message ends and the next begins.

This lesson shows the three framing styles you will meet most often (delimiters, length prefixes and fixed sizes), how asyncio's reader helps with each, and what errors to expect when the data is damaged or too large. Every example feeds bytes into a StreamReader by hand, so you can see exactly what arrives and in which pieces.

Style 1: delimiter framing (lines)

The simplest rule is "a message ends at a newline". HTTP headers, SMTP, Redis' old inline protocol and the JSON-lines format all work this way. readline() is built for it.

Python Newline-delimited JSON

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first piece contains half a message. readline() simply keeps waiting until it sees the newline, so your code never has to glue pieces together. The cost of this style is that the payload itself must never contain the delimiter. JSON is safe because a newline inside a string is escaped as \n.

readuntil(separator) generalises this to any byte string, such as b"\r\n\r\n" for the end of HTTP headers or b"\0" for a NUL-terminated protocol.

Style 2: length-prefixed frames

For binary data, where any byte value may appear in the payload, put the size first. The receiver reads the fixed-size header, learns the length n, and then calls readexactly(n). This is how TLS records, PostgreSQL, Kafka, gRPC and countless custom protocols work.

flowchart LR
    A["4 bytes<br/>length N"] --> B["N bytes<br/>payload"]
    B --> C["next frame..."]

A frame is a fixed header giving the payload size, followed by exactly that many bytes.

The struct module packs and unpacks the header. ">I" means a big-endian ("network order") unsigned 32-bit integer.

Python Length-prefixed frames

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two details matter here. An empty payload is legal (the length is zero and readexactly(0) returns b"" immediately). And when the stream ends cleanly between frames, readexactly() raises IncompleteReadError with an empty partial; that is how you detect a normal disconnect.

Style 3: fixed-size records

If every message is, say, exactly 16 bytes, just call readexactly(16) in a loop. This is rare on its own, but it is the foundation of the length prefix (a fixed-size header).

Errors you must handle

IncompleteReadError

readexactly(n) and readuntil(sep) raise asyncio.IncompleteReadError when the stream hits end of file first. Its .partial attribute holds what did arrive, and .expected the number of bytes you asked for. If partial is non-empty, the peer disconnected in the middle of a message: log it and drop the connection.

LimitOverrunError and ValueError

A reader protects your memory by limiting how much readline() and readuntil() will buffer (the limit argument, 64 KiB by default). A line that exceeds the limit makes readline() raise ValueError, while readuntil() raises asyncio.LimitOverrunError. An unbounded line from a malicious client must never be able to fill your memory, so treat these errors as "close the connection".

Python Limits and incomplete reads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a framing

  • Text protocols that humans debug: newline or \r\n delimiters. Easy to try with telnet or nc.
  • Binary payloads or large messages: length prefix. Always cap the maximum length you accept (for example reject anything above a few megabytes) before calling readexactly(length), or a hostile client can ask you to allocate gigabytes.
  • Mixed: a text header terminated by a blank line, followed by a body whose size the header announces. That is HTTP.

Whatever you choose, write the encoding and decoding as two small functions, test them with feed_data as above, and use them on both client and server. The TCP echo lesson puts a framed protocol on a real server shape.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A client sends two messages with two separate writes. What can the server's reader see?

    TCP is a byte stream. Chunks can be merged or split arbitrarily, which is why protocols need framing.
  2. 2. Why does a length-prefixed protocol need a maximum frame size on the receiver?

    The length comes from the network, so it is untrusted. Check it before reading the payload.
  3. 3. What does IncompleteReadError.partial contain?

    partial holds what arrived before end of file; expected holds how many bytes were requested.
  4. 4. Which framing is the safest for arbitrary binary payloads?

    A length prefix never depends on the content of the payload, so any byte value can appear inside it.

Summary

  • A TCP stream has no message boundaries; your protocol must define them.
  • readline() and readuntil() handle delimiters; readexactly() plus a struct header handles length-prefixed frames.
  • IncompleteReadError means the peer disconnected (check .partial); size limits protect your memory.
  • Validate lengths from the network before allocating or reading.

Next: A TCP echo server and client