Skip to content
Tutorials Center
My learning

Topics

Lesson 88 of 132 ยท Standard Library Tour

Read, Write and BufRead Traits

Master Rust's I/O traits: write code generic over Read, Write and BufRead, use Cursor and Vec<u8>, lock stdout, implement your own Write adapter, and parse bytes.

A file, a network socket, the keyboard, a byte array in memory and a compressed stream are very different things, yet reading from them looks the same: ask for some bytes, get some bytes. Rust captures that with three small traits in std::io: Read (a source of bytes), Write (a destination for bytes) and BufRead (a source with a buffer, which adds lines and delimiters). Once your code is written against these traits, it works with files today, with a TCP stream tomorrow, and with a plain Vec<u8> in a unit test.

This is one of the most useful ideas in the standard library, because it is just traits and generics applied to I/O. In this lesson you will write functions that accept any reader or writer, use Cursor and Vec<u8> as in-memory streams, copy data between streams, lock stdout properly, implement Write yourself to build adapters, and read binary data. It builds on files and the filesystem.

The three traits in one minute

  • Read has one required method, read(&mut self, buf: &mut [u8]) -> io::Result<usize>, which fills part of the buffer and returns how many bytes it wrote. 0 means end of stream. Everything else is built on it: read_to_string, read_to_end, read_exact, bytes, take, chain.
  • Write has two required methods, write(&mut self, buf: &[u8]) -> io::Result<usize> and flush(). Built on them: write_all and the write!/writeln! macros.
  • BufRead extends Read with an internal buffer and adds read_line, lines, read_until and split.

Note the word some: a single read call may return fewer bytes than the buffer holds, even when more data is coming. That is why the helpers read_exact (insist on filling the buffer) and write_all (insist on writing everything) exist. You will almost always use those instead of the raw methods.

Writing code that is generic over streams

Compare two ways to write a function that counts the words in some text. The first only works with files. The second works with every reader that can be buffered:

Rust One function, many sources

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

&[u8] implements Read and BufRead directly, which makes it perfect for tests: no files needed. The function never says where its data comes from. In a real program you would pass BufReader::new(File::open(path)?) or io::stdin().lock(), and the function would not change.

The same works for output. A function that writes its report to a Write can print to the terminal, go to a file, or fill a Vec<u8> that a test inspects:

Rust Generic output with Write

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

io::sink() is a writer that discards everything, handy for benchmarks and for turning off output; io::empty() is a reader that is always at end of stream; io::repeat(b'x') is an endless reader of one byte.

Take the writer as &mut W or W: Write rather than returning a String. The caller decides where the text goes, and large outputs are never built in memory. Because &mut W also implements Write when W does, you can pass &mut file to a function and keep using file afterwards.

A very common compile error

write_all, read_to_string and friends are trait methods. If the trait is not imported, the method is invisible, and the error message can be confusing the first time:

Rust Forgetting to import the trait (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

rustc reports error[E0599]: no method named write_allfound for structVec in the current scope, and then helps: items from traits can only be used if the trait is in scope, with the suggestion use std::io::Write;. Add that line at the top and the program prints [104, 101, 108, 108, 111]. The same applies to Read, BufRead, Seek and fmt::Write.

There are two different traits called Write: std::io::Write (bytes, can fail with io::Error) and std::fmt::Write (text, used to format into a String). The macros write! and writeln! work with both, but they are different traits. If you import both, rename one: use std::fmt::Write as _;.

Cursor: an in-memory file

io::Cursor<T> wraps a Vec<u8>, a &[u8] or similar, and adds a position, so the memory behaves like a file: you can read from it, write to it and, since it implements Seek, jump around. It is the standard tool for testing I/O code without touching the disk.

Rust Cursor with read, write and seek

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Note the difference at the end of the stream: read returns Ok(0) (not an error) and read_exact returns the error UnexpectedEof with the message failed to fill whole buffer. A loop that reads until Ok(0) is the classic way to consume a stream.

Copying, limiting and chaining

Because every reader and writer speaks the same language, generic helpers exist for combining them:

  • io::copy(&mut reader, &mut writer) moves everything and returns the byte count. It is how you copy a file or forward a socket to a file with one call.
  • reader.take(n) limits a reader to its first n bytes. It protects you from an attacker who sends an endless stream.
  • a.chain(b) reads a until the end and then b.
  • reader.bytes() yields one byte at a time as io::Result<u8>. It is simple but slow unless the reader is buffered.
  • BufRead::read_until(delim, &mut vec) and split(delim) cut a stream at any byte, not just newlines.
Rust copy, take, read_until and split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Locking stdout (and stdin)

println! locks standard output for every call, prints, and unlocks. That is fine for a few lines, but a loop of 100,000 println! calls takes the lock and (when stdout is a terminal or pipe with line buffering) performs a write per line. The faster approach is to lock once and write through a BufWriter:

Rust
let stdout = io::stdout();
let mut out = io::BufWriter::new(stdout.lock());
for n in 1..=100_000 {
    writeln!(out, "{n}")?;
}
out.flush()?;

The same idea applies to input: io::stdin().lock() gives a BufRead, so you can pass it to any generic function that takes a BufRead. The next program reads numbers from the Input box with the generic function from the beginning of this lesson, and writes its answer through a locked, buffered stdout.

Rust Generic input from stdin, buffered output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

With the preset input the program reports ignoring "x" for line 2 and prints a total of 172. Replace the input with your own lines to experiment. The lock guards live as long as the variables that hold them, and sum_numbers uses the buffered out for its warnings, so there is exactly one owner of each stream. Keep it that way: if you also called println! while out holds unflushed data, the two outputs would appear in a surprising order.

Implementing Write yourself: adapters

Writing your own Write is easy and powerful, because you can wrap another writer and change what passes through. Two required methods: write and flush. Below, Counting counts bytes, Upper converts ASCII letters to uppercase on their way through, and Tee sends everything to two destinations. They stack like building blocks.

Rust Writing your own Write adapters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Look at Counting::write: it counts n, the number returned by the inner writer, not buf.len(). A writer is allowed to accept only part of the buffer, and the caller (write_all) will call again with the rest. Getting that wrong is a classic bug in hand-written adapters. Upper and Tee use write_all on the inner writer and then report that everything was consumed, which is also correct. This wrap-and-forward design is exactly how BufWriter, compression encoders and TLS streams are built. Implementing Read works the same way, by filling the caller's buffer from an inner reader and changing the bytes on the way.

Byte processing: parsing a binary format

Real binary data has fixed-size fields. Use read_exact into small arrays and convert them with the from_be_bytes, from_le_bytes and from_ne_bytes methods that every integer type has. The tiny format below has a 4-byte magic number, a big-endian u16 count, and then count records, each with a length byte followed by that many UTF-8 bytes.

Rust Reading a length-prefixed binary format

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

decode takes any Read, so it works on a file, a socket or the Cursor used here. When parsing network data, never trust a length field blindly: a length of 4 billion followed by vec![0; len] would allocate gigabytes. Check it against a sane maximum, or read with take(len) and read_to_end.

Quick reference

Task Tool
Read everything into a String / Vec<u8> read_to_string, read_to_end
Exactly N bytes read_exact (error: UnexpectedEof)
Line by line BufRead::lines, read_line
Up to a delimiter read_until, split
Write everything write_all, write!, writeln!
Buffer small reads/writes BufReader, BufWriter
Test with memory instead of files &[u8], Cursor, Vec<u8>
Copy a whole stream io::copy
Throw output away / empty input io::sink(), io::empty()

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does Read::read return when the stream has ended?

    Ok(0) signals end of stream. Only read_exact turns a short read into the UnexpectedEof error.
  2. 2. Why does the compiler say 'no method named write_all found' for a Vec<u8>?

    Vec<u8> does implement io::Write, but trait methods are only available when the trait is in scope. Add use std::io::Write;
  3. 3. Which signature lets a function accept a file, a socket, stdin or a byte slice?

    A generic parameter bounded by Read accepts every type that implements the trait. This also makes the function easy to test with in-memory data.
  4. 4. In a custom Write adapter, what should write() return?

    write may accept only part of the buffer. The caller (write_all) uses the returned count to continue with the rest. If you forward with write_all and consumed everything, returning buf.len() is correct.
  5. 5. Why is a BufWriter around a locked stdout faster than many println! calls in a loop?

    Fewer system calls and one lock instead of many. Remember to flush the BufWriter at the end.

Summary

  • Read, Write and BufRead describe sources and destinations of bytes. Write your functions against these traits (or &mut dyn Write) so they work with files, sockets, stdin/stdout and in-memory data.
  • &[u8], Vec<u8> and Cursor are in-memory readers and writers: perfect for tests. Cursor also supports Seek.
  • Prefer read_exact and write_all over the raw read/write, which may handle only part of the buffer. Ok(0) from read means end of stream.
  • Trait methods need the trait in scope (use std::io::Write;). Do not confuse io::Write with fmt::Write.
  • Helpers combine streams: io::copy, take, chain, bytes, read_until, split, io::sink, io::empty.
  • Lock stdout once and wrap it in a BufWriter for heavy output; flush at the end.
  • Implement Write (and Read) to build adapters that wrap another stream, and parse binary data with read_exact plus from_be_bytes/from_le_bytes, validating lengths.

Next: Environment, Processes and Signals