Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 40 ยท Networking and Streams

Protocols and Transports (Low-Level asyncio)

Learn asyncio's low-level Protocol and Transport API: connection_made, data_received, connection_lost, create_server, and when to prefer streams instead.

Streams are built on top of a lower layer made of two cooperating objects: a transport, which knows how bytes move (a TCP socket, a TLS session, a pipe), and a protocol, which knows what the bytes mean. The event loop calls methods on your protocol when things happen. This is the callback style that asyncio started with, and it is still the fastest option because there is no extra task or await per message.

Most application code should use streams. You reach for protocols when you need maximum throughput, when a library hands you a protocol class to subclass, or when you want to understand what streams are doing underneath.

The division of labour

  • The transport is created by the event loop. You call it: transport.write(data), transport.close(), transport.get_extra_info("peername"), transport.pause_reading().
  • The protocol is yours. The loop calls it: connection_made, data_received, eof_received, connection_lost.
flowchart LR
    S["Socket / network"] --> T["Transport"]
    T -->|"data_received(data)"| P["Your Protocol"]
    P -->|"transport.write(data)"| T
    T --> S

Bytes flow in through callbacks on the protocol and out through method calls on the transport.

An echo protocol

Python
class EchoProtocol(asyncio.Protocol):
    def connection_made(self, transport):
        self.transport = transport

    def data_received(self, data):
        self.transport.write(data)

    def connection_lost(self, exc):
        pass            # exc is None for a clean close, an exception otherwise

On a real machine you start it with loop.create_server(EchoProtocol, "127.0.0.1", 8888) (one protocol instance per connection, because the first argument is a factory) and connect with loop.create_connection(factory, host, port). In the browser we cannot create sockets, but a protocol is just a class: we can give it a fake transport and call its callbacks ourselves, which is also how you unit-test protocols.

Python Driving a protocol by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The callbacks in order

  1. connection_made(transport): called once, first. Store the transport.
  2. data_received(data): called zero or more times, each time with whatever bytes arrived. The chunks have arbitrary boundaries, so framing applies here exactly as before. You must handle partial messages yourself.
  3. eof_received(): the peer signalled it will send no more (half-close). Return True to keep the transport open so you can still write; the default (returning a falsy value) closes it.
  4. connection_lost(exc): called once, last, with None for a normal close or the exception for an error.

Two rules for callbacks: they are plain functions, not coroutines, so you cannot await inside them, and they must return quickly because they run on the event loop. If you need to await something (a database call, say), start a task with asyncio.create_task() from inside the callback and keep a reference to it.

A line-based protocol with a buffer

Because data_received hands you arbitrary chunks, a real protocol keeps a buffer. This one splits the input into lines and replies to each:

Python Buffering partial lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The on_con_lost future is the standard way for a client program to wait until the connection ends: the protocol sets a result in connection_lost, and the main coroutine awaits it. It is how Python's own documentation examples finish cleanly.

Protocol or streams?

Streams (StreamReader/StreamWriter) Protocols
await based, reads like normal code callback based, state lives in attributes
One task per connection no task per connection, lowest overhead
Easy timeouts, cancellation and error handling you manage everything yourself
Best for application code best for libraries and hot paths

Streams are in fact a thin layer: StreamReaderProtocol is a protocol that calls reader.feed_data() in data_received, and StreamWriter wraps the transport. That is exactly what the in-memory helper in the streams lesson used.

Other protocol flavours

  • asyncio.BufferedProtocol: the loop asks you for a buffer (get_buffer) and writes straight into it (buffer_updated), avoiding a copy per chunk. Useful for high-throughput servers.
  • asyncio.DatagramProtocol: for UDP, with datagram_received(data, addr) and loop.create_datagram_endpoint().
  • asyncio.SubprocessProtocol: output of child processes via loop.subprocess_exec().
  • Flow control callbacks pause_writing() and resume_writing() also live here, as explained in flow control.

Common mistakes

  • Passing an instance instead of a factory to create_server(). Every connection needs its own protocol object, so pass the class or a lambda.
  • Doing slow or blocking work in data_received, which stalls the whole loop.
  • Assuming one data_received call is one message.
  • Using the transport after connection_lost: writes to a closed transport are ignored or logged as warnings.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the Protocol API, who calls data_received()?

    The loop invokes protocol callbacks; your code calls methods on the transport.
  2. 2. Why does loop.create_server() take a protocol factory instead of one protocol instance?

    Per-connection state such as buffers lives on the protocol, so each accepted connection gets a fresh instance from the factory.
  3. 3. Can data_received contain "await asyncio.sleep(1)"?

    Protocol callbacks are called synchronously by the loop. Use create_task() for asynchronous follow-up work.
  4. 4. What does returning True from eof_received() do?

    Returning a true value keeps the transport open after the peer's half-close, so you can still send a reply.

Summary

  • A transport moves bytes; a protocol decides what they mean. The loop calls the protocol's callbacks, and you call the transport's methods.
  • The callback sequence is connection_made, any number of data_received, optionally eof_received, then connection_lost.
  • Callbacks are synchronous: keep them fast, buffer partial messages, and use create_task() for async work.
  • Prefer streams for application code and protocols for libraries and performance-critical paths.

Next: Flow control and backpressure