Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 40 ยท Networking and Streams

Streams - StreamReader and StreamWriter

Learn asyncio streams: how StreamReader and StreamWriter work, how to read bytes, lines and chunks, write with drain, and close a connection cleanly in Python.

Most asyncio programs spend their time waiting for something other than a timer: bytes arriving over a network connection. asyncio's streams are the high-level, async/await-friendly way to talk over TCP, Unix sockets, TLS and pipes. You get two small objects, a StreamReader to read from and a StreamWriter to write to, and you never touch a raw socket.

This lesson explains what those two objects do and which methods you will use every day. Because the lesson editor runs in your browser, where raw sockets are not available, the examples use a tiny in-memory connection that behaves like a real one. The code that uses the streams is exactly the code you would write for a real socket.

About the examples. In the browser sandbox asyncio.open_connection() and asyncio.start_server() raise NotImplementedError, because browsers do not allow raw sockets. The helper called memory_connection() builds a real StreamReader and StreamWriter pair joined by a pipe in memory. Run the same handler functions against real sockets on your own computer and they behave the same way. The helper is shown in full below so you can see there is no magic.

The two halves of a connection

A connection has a receiving side and a sending side, and asyncio models them as separate objects:

  • StreamReader has coroutine methods such as read(), readline(), readexactly() and readuntil(). Each one waits (without blocking the event loop) until enough bytes have arrived.
  • StreamWriter has plain methods like write() and close(), plus two coroutines: drain() and wait_closed().
Python
reader, writer = await asyncio.open_connection("example.org", 80)

open_connection() is a coroutine that connects and returns the pair. On the server side, asyncio.start_server() calls your handler with a fresh pair for every client. You will build both in the TCP echo lesson.

flowchart LR
    A["Your code"] -->|"writer.write(data)"| B["StreamWriter"]
    B -->|"bytes"| C["Network"]
    C -->|"bytes"| D["StreamReader"]
    D -->|"await reader.read()"| A

Your coroutine writes into the writer and reads out of the reader; the event loop moves the bytes in between.

A first conversation

Here is the helper and a complete exchange. Read the helper once: a StreamWriter needs a transport to send bytes through, and ours simply hands them to the other side's reader.

Python Two ends of a connection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice three habits that appear in every stream program: write() is called without await, drain() follows it, and reads are always awaited.

Reading: five methods, five jobs

Bytes on a network have no built-in boundaries. A single send on one side can arrive as two chunks, and two sends can arrive glued together. The reader methods exist to put boundaries back:

Method Returns Use it when
await reader.read(n) up to n bytes (whatever is available, at least one) you process data in chunks
await reader.read() everything until the other side closes you expect one finite message
await reader.readline() bytes up to and including \n the protocol is line based
await reader.readexactly(n) exactly n bytes or an error fixed-size headers and payloads
await reader.readuntil(sep) bytes up to and including a separator custom delimiters

At end of stream (the peer closed or called write_eof()), read(n) and readline() return b"". That empty value is your loop-exit signal.

Python Reading the same bytes several ways

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

feed_data() and feed_eof() are how asyncio itself delivers incoming bytes into a reader. You rarely call them in application code, but they are perfect for experiments and for tests, as you will see in testing async code.

Writing and draining

writer.write(data) does not wait. It appends the bytes to an internal buffer, and the event loop sends them when the socket is ready. That is why it is not a coroutine, and also why you must not call it a million times in a tight loop without pausing: the buffer would grow without limit.

await writer.drain() is the pause. It returns at once while the buffer is small, and suspends your task when the buffer is above its high-water mark, until the peer has read enough. Use the pair as a habit:

Python
writer.write(chunk)
await writer.drain()

The lesson on flow control and backpressure shows what drain() is protecting you from.

writer.writelines(list_of_bytes) writes several pieces at once, and writer.write_eof() tells the peer "I am done sending" while keeping your reading side open (a half-close).

Closing correctly

Closing is two steps, because the actual shutdown happens later in the event loop:

Python
writer.close()
await writer.wait_closed()

Skipping wait_closed() can leave buffered bytes unsent or print "unclosed transport" warnings at exit. In a handler that might fail, put the pair in try/finally.

Python Half-close with write_eof

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Useful extras

  • writer.get_extra_info("peername") returns the remote address on a real socket ("sockname", "ssl_object" and others exist too).
  • reader.at_eof() is true when no more data will ever arrive.
  • asyncio.start_tls() upgrades an existing connection to TLS, and open_connection(..., ssl=True) connects with TLS from the start.
  • asyncio.open_unix_connection() and start_unix_server() do the same over Unix domain sockets.
  • A reader has a size limit (64 KiB by default) for readline() and readuntil(); the next lesson covers what happens when a line is longer.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these StreamWriter methods is a coroutine that you must await?

    write(), close() and write_eof() are plain methods that return immediately. drain() may need to wait for the peer, so it is a coroutine.
  2. 2. What does reader.read() return when the peer has closed the connection and everything was already read?

    End of stream is reported as b"". Loops like "while data := await reader.read(n)" rely on it.
  3. 3. Why call "await writer.wait_closed()" after "writer.close()"?

    close() only starts the shutdown. wait_closed() returns once it is complete, so buffered data is sent and resources are released.
  4. 4. In the browser sandbox, what happens if you call asyncio.open_connection("example.org", 80)?

    Browsers do not allow raw sockets, so the lesson examples use an in-memory connection with the same reader and writer classes.

Summary

  • A connection is a StreamReader (awaitable reads) plus a StreamWriter (buffered writes).
  • Use readline(), readexactly() and readuntil() to restore message boundaries; b"" means end of stream.
  • write() buffers, await drain() applies backpressure, close() then await wait_closed() shuts down cleanly.
  • feed_data() lets you build readers from bytes you already have, which is ideal for experiments and tests.

Next: Framing messages over a stream