Skip to content
Tutorials Center
My learning

Topics

Lesson 78 of 147 ยท Standard Library

os, io and bufio

Learn Go's io.Reader and io.Writer, helpers like Copy, TeeReader, MultiWriter and Pipe, buffered I/O with bufio.Scanner and Writer, and the os package for arguments, environment, files and exit codes.

Reading a file, receiving an HTTP request, compressing data, hashing a stream and printing to the terminal all look different, but Go treats them with the same two tiny interfaces: io.Reader and io.Writer. Because so many types implement them, you can connect almost any source to almost any destination, and you can build pipelines out of small parts, the way Unix tools connect with pipes.

This lesson covers the two interfaces, the helper functions in io that combine them, the bufio package that makes I/O faster and more convenient, and the os package for the things a program gets from its surroundings: command-line arguments, environment variables, standard streams, small files and exit codes. Working with directories and paths comes next in files, directories and paths.

The two interfaces

Both interfaces have exactly one method:

Go
type Reader interface {
	Read(p []byte) (n int, err error)
}

type Writer interface {
	Write(p []byte) (n int, err error)
}

Read fills the slice p with up to len(p) bytes and says how many it put there. It returns io.EOF when there is no more data. Write takes all of p and must return an error if it wrote fewer bytes than that. Three rules about Read are worth memorizing, because they explain a lot of bugs:

  • Process n bytes before looking at err. A reader may return n > 0 and an error (even io.EOF) in the same call.
  • A short read is normal. Getting 3 bytes back for a 4096-byte buffer does not mean the end of the data. Keep reading until io.EOF.
  • io.EOF is a signal, not a failure. It is the sentinel error described in errors as values and is the only error that means "finished cleanly".

You rarely call Read and Write by hand. You implement them on your own types, or hand readers and writers to functions that use them. Here is a reader that wraps another reader and changes the text on the fly, plus a writer that only counts bytes. Both plug straight into io.Copy, which moves data from a reader to a writer until the reader reports io.EOF:

Go Writing your own Reader and Writer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

io.Copy returns the number of bytes copied (39 here) and nil at a normal end. It uses a 32 KB buffer internally, so it handles a 10 GB file as easily as a 10 byte string without holding it in memory. Notice that fmt.Fprintf accepts any io.Writer, which is why the fmt functions work with files, network connections, buffers and your own types alike.

The standard library is full of types that implement these interfaces:

Type Reader Writer Typical use
*os.File yes yes files, os.Stdin, os.Stdout, os.Stderr
*strings.Reader yes no read from a string (great in tests)
*bytes.Buffer yes yes build or collect bytes in memory
*strings.Builder no yes build a string efficiently
net.Conn yes yes network connections
http.ResponseWriter / http.Request.Body writer / reader HTTP
*bufio.Reader / *bufio.Writer yes / no no / yes buffering
io.Discard no yes throw output away

Combining readers and writers

The io package has small building blocks that take readers or writers and return new ones. Because the inputs and outputs are interfaces, they snap together:

Function What it does
io.ReadAll(r) reads until EOF and returns everything as a []byte
io.Copy(w, r) / io.CopyN(w, r, n) copy everything / copy exactly n bytes
io.LimitReader(r, n) a reader that stops after n bytes
io.MultiReader(r1, r2, ...) reads r1 to its end, then r2, and so on
io.MultiWriter(w1, w2, ...) a writer that duplicates every write to all of them
io.TeeReader(r, w) a reader that also writes everything it reads to w
io.NewSectionReader(ra, off, n) a reader for a slice of a file or other ReaderAt
io.ReadFull(r, buf) fills buf completely, or fails with io.ErrUnexpectedEOF
io.Pipe() a connected reader and writer in memory, for goroutines
Go Tee, Multi, Limit and Pipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A few things to learn from it. The tee example is the "read once, use twice" pattern that appears all the time when downloading and verifying a file in one pass. io.ReadFull distinguishes two cases: EOF when no bytes were read at all, and unexpected EOF when the data ended in the middle of the buffer. And io.Pipe is synchronous and unbuffered: the writer blocks until a reader takes the data, so the two ends must run in different goroutines (see goroutines). Always close the write end, or the reader waits forever.

io.ReadAll loads everything into memory. Never call it on data you do not control, such as an HTTP request body, without limiting it first: io.ReadAll(io.LimitReader(r, 1<<20)) reads at most 1 MiB.

Buffered I/O with bufio

Each call to Read or Write on a file or a network connection can be a slow system call. If your program reads a byte at a time or prints many tiny strings, it spends most of its time in the operating system. The bufio package wraps a reader or writer with an in-memory buffer, so the real I/O happens in large chunks. It also adds convenient ways to read text.

bufio.Scanner: the easy way to read lines

A Scanner reads input in pieces called tokens. By default a token is a line without its line ending:

Go Reading lines and words with bufio.Scanner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The empty line shows up as "": a blank line is still a line. Scan returns false at the end of input or on an error, so check sc.Err() after the loop. The other split functions are bufio.ScanLines (the default), ScanWords, ScanRunes and ScanBytes, and you can write your own.

A scanner has a maximum token size of 64 KB by default. A longer line stops the scan with bufio.Scanner: token too long. If you must read longer lines, enlarge the limit with Buffer:

Go The Scanner token limit and bufio.Reader

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

bufio.Reader.ReadString('\n') keeps the delimiter and returns whatever is left (together with io.EOF) when the input does not end in a newline, so process the returned text before you check the error. Use Scanner for typical line-by-line text, and Reader for binary-ish formats, very long lines, or when you need Peek and ReadRune. You can also meet fmt.Fscan in fmt in depth when you want to parse numbers.

bufio.Writer: batch your output

A bufio.Writer collects what you write and sends it on when its buffer (4096 bytes by default) is full or when you call Flush. If you forget Flush, the last part of your output is lost.

Go A buffered writer must be flushed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The unbuffered line is printed before the two buffered ones even though it was written later: the buffered text only leaves when the deferred Flush runs. Buffering matters for programs that print hundreds of thousands of lines, such as competitive-programming solutions, where wrapping os.Stdout in a bufio.Writer can make the output step many times faster.

The os package

os gives your program access to the operating system. These are the parts you will use first.

Standard streams and arguments

os.Stdin, os.Stdout and os.Stderr are *os.File values, so they are readers and writers. Normal output goes to stdout, and diagnostics and errors go to stderr, so users can redirect one without the other (prog > out.txt captures only stdout).

os.Args is a []string with the program name at index 0 and the command-line arguments after it. For anything beyond one or two arguments use the flag package. Our editor runs your code without arguments, so os.Args[1:] is empty here.

Environment variables

os.Getenv("NAME") returns the value, or an empty string if it is unset. When you must tell "unset" from "set but empty", use os.LookupEnv, which returns a second ok value. os.Setenv, os.Unsetenv, os.Environ() and os.ExpandEnv("$NAME") complete the set.

Whole files

For small files, os.ReadFile and os.WriteFile read and write the entire content in one call, and they open and close the file for you. The error values wrap *fs.PathError, and you can test them with errors.Is(err, fs.ErrNotExist).

Go Arguments, environment, files and stderr

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The editor shows stderr separately from stdout, just as a terminal would let you redirect them separately. 0o644 is the file permission in octal (owner can read and write, everyone else only read); more on permissions in the next lesson.

Exit codes

When a program ends, it reports a number to the shell: 0 means success, anything else means failure. Returning from main exits with 0. To exit with another code, call os.Exit(code). Two warnings go with it:

  1. os.Exit ends the program immediately: deferred functions do not run. Buffers that you meant to flush with defer are lost.
  2. A panic that is not recovered exits with code 2.

The way to combine clean-up with exit codes is to keep main tiny and put the logic in a function that returns an error, so that all defers run before you exit:

Go os.Exit skips deferred calls; the run() pattern

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

You see the cleanup line from run but not the deferred line in main. In a terminal you can read the exit status of the last command:

Terminal
$ go run main.go --bad-flag
error: no arguments expected
exit status 1
$ echo $?
1

(On Windows PowerShell the status is in $LASTEXITCODE.) The same convention is used by log.Fatal, which prints a message and calls os.Exit(1); see logging and panic and recover.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A call to Read(p) returns n = 5 and err = io.EOF. What should the caller do?

    A Reader may return data together with io.EOF. Always handle the n bytes first, then look at the error. io.EOF just means the data has ended.
  2. 2. What does io.TeeReader(r, w) do?

    Whatever is read through the tee is also written to w. Combined with io.MultiWriter, it lets you store and hash a stream in one pass.
  3. 3. A program writes with w := bufio.NewWriter(os.Stdout) and never calls w.Flush(). What happens to short output?

    A bufio.Writer sends data on only when its buffer is full or when Flush is called. Without Flush the last part of the output is lost.
  4. 4. Why does bufio.Scanner fail on a 100,000 character line by default?

    Scan returns false and Err reports bufio.Scanner: token too long. Call scanner.Buffer to allow larger tokens, or read with bufio.Reader.
  5. 5. What happens to defer statements in main when you call os.Exit(1)?

    os.Exit terminates the process immediately without running deferred functions. Put your logic in a function that returns an error and call os.Exit only from main.

Summary

  • io.Reader and io.Writer are one-method interfaces implemented by files, buffers, network connections and more. Handle the n bytes returned by Read before checking its error, and treat io.EOF as a normal end.
  • io.Copy, ReadAll, LimitReader, MultiReader, MultiWriter, TeeReader, SectionReader, ReadFull and Pipe combine readers and writers into pipelines. Limit untrusted input before ReadAll.
  • bufio.Scanner reads lines, words or runes (64 KB token limit, Buffer to raise it); bufio.Reader has ReadString and Peek; bufio.Writer batches output and needs Flush.
  • os.Args, os.Getenv/LookupEnv, os.ReadFile/WriteFile and the three standard streams connect the program to its environment. Send errors to os.Stderr.
  • Exit code 0 means success. os.Exit skips deferred calls, so keep main small and call it only after a run() error function has returned.

Next: Files, Directories and Paths goes further with opening files, walking directories and building portable paths.