Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 147 ยท Collections

Bytes, Buffers and Readers

Learn Go byte slices, the bytes package, bytes.Buffer, strings.Reader, bufio.Reader and Writer, and the io.Reader and io.Writer interfaces that let you stream and build data efficiently.

A string is immutable text. A byte slice ([]byte) is the same data in a form you can change: overwrite a byte, append more, reuse the buffer. Most input and output in Go, from files to network sockets to HTTP bodies, is made of bytes. So Go has a small toolkit built around them: the bytes package (the twin of strings), the bytes.Buffer type for building and consuming data, and two tiny interfaces, io.Reader and io.Writer, that tie everything together.

This lesson introduces that toolkit gently. It's a preview: the os, io and bufio lesson goes deeper into composing readers and writers, and Files, Directories and Paths applies it to real files.

Byte slices and strings

[]byte is just a slice of byte (an alias for uint8). Converting between a string and a byte slice copies the data in both directions, because strings must stay immutable:

Go Converting between string and []byte

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Notice append(b, " there"...): the special form of append that accepts a string as its variadic source for a byte slice. The last two lines remind you that len counts bytes, not characters; see Strings, Bytes and Runes.

The copy has a cost for big data. If a hot path converts the same large string over and over, keep it as []byte the whole time, or avoid the conversion by using functions that accept strings directly. Don't reach for unsafe tricks to avoid the copy until a profile proves it matters.

The bytes package

bytes mirrors strings function by function, working on []byte instead of string: Contains, Index, HasPrefix, Split, Fields, Join, Trim, ToUpper, Replace, Cut and more. There are two notable differences: you compare slices with bytes.Equal (not ==), and bytes.Compare orders them:

Go The bytes package mirrors strings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The functions that return slices (Split, Trim, Cut) return views into the original data rather than copies. That makes them fast, but a change to one is visible in the other, the same aliasing rule as How Slices Work described.

bytes.Buffer: a growable byte container

bytes.Buffer is a variable-sized buffer that you can write to and read from. Its zero value is ready to use, so you don't need a constructor. It's the byte-oriented sibling of strings.Builder from Working with Strings, with one extra power: it can also be read from.

Go Writing to a bytes.Buffer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

buf.String() copies the unread bytes into a new string. buf.Bytes() returns a slice that aliases the buffer's own storage, valid only until the next write or read, so copy it if you need to keep it.

Reading from a Buffer

A buffer works like a queue: you write at the end and read from the front, and reading consumes what you read:

Go Reading consumes a Buffer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

When a reader has nothing left, it returns the special error io.EOF. That value means "the normal end", not a failure; Errors as Values explains why Go uses error values for it.

io.Reader and io.Writer

Notice what bytes.Buffer, strings.Builder, os.Stdout, files and network connections share: you can write bytes to them. Go captures that with two one-method interfaces from the io package:

Go
type Writer interface {
	Write(p []byte) (n int, err error)
}

type Reader interface {
	Read(p []byte) (n int, err error)
}

Anything with those methods is a Writer or Reader, with no declaration needed (that is how interfaces work in Go). Functions that take these interfaces work with files, memory, the network and test fakes alike. fmt.Fprintf(w, ...) accepts any io.Writer, which is why the same formatting code can write to the screen, a buffer or a file:

Go One function, many destinations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

If you run this in the editor, the os.Stderr line appears in the error output rather than the regular output. Read on for the reading side.

strings.Reader and io helpers

strings.NewReader(s) turns a string into an io.Reader (and bytes.NewReader does the same for a []byte). That lets you feed text into any function that expects a reader, which is exactly what you want in tests and examples. The io package provides helpers: io.ReadAll reads everything, io.Copy moves data from a reader to a writer in chunks, and io.LimitReader stops after a number of bytes:

Go strings.Reader, io.ReadAll and io.Copy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A single Read call returns "up to" len(p) bytes, and it may return fewer. The correct loop processes the n bytes you got even when an error comes back at the same time, which is why the example prints chunk[:n] before looking at err. Use io.ReadAll and io.Copy when you can; they handle all the edge cases.

Buffered I/O with bufio

Each call to Write on an unbuffered file or socket may cause a slow system call. The bufio package wraps a reader or writer with an in-memory buffer so that many small operations become a few big ones. bufio.Scanner reads lines or words, bufio.Reader gives finer control, and bufio.Writer collects output until you Flush it:

Go bufio.Scanner and bufio.Writer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The two golden rules of bufio.Writer: call Flush before the program exits (the defer takes care of that), and don't mix it with direct writes to the same destination. This pattern, a buffered writer on os.Stdout, is the standard trick for programs that print many lines quickly. More on the input side is in Reading Input.

Building output efficiently

A quick guide to choosing the right tool:

You want to... Use
Build a string from pieces strings.Builder
Build bytes and also read them back bytes.Buffer
Print many lines to the screen or a file bufio.NewWriter + Flush
Treat a string as a stream strings.NewReader
Treat a []byte as a stream bytes.NewReader or bytes.NewBuffer
Read everything from a source io.ReadAll
Move data between a reader and a writer io.Copy

For a final example, here is a tiny program that reads a stream in chunks and counts lines without ever holding the whole input in memory, using only the interfaces from this lesson:

Go Counting lines from any reader

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

countLines doesn't know or care where the data comes from. It could just as well be a file, an HTTP response or a decompressor, and that is the main reason Go programmers love io.Reader.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does b := []byte("hi"); b[0] = 'H'; fmt.Println(string(b)) print?

    The conversion copies the bytes into a new slice, which can be modified freely. The original string would stay unchanged.
  2. 2. Which function compares the contents of two []byte values?

    Slices cannot be compared with ==, so use bytes.Equal. (Only a comparison with nil is allowed with ==.)
  3. 3. What is the value of var buf bytes.Buffer before any use?

    The zero value of bytes.Buffer is an empty buffer, so no constructor is needed.
  4. 4. Why must you call Flush on a bufio.Writer?

    A bufio.Writer keeps data in memory until the buffer is full or you flush it. Without a final Flush, the last part may never be written.
  5. 5. What does a Read that returns io.EOF mean?

    io.EOF signals the end of the stream. It is expected, not a failure.

Summary

  • A []byte is a mutable sequence of bytes. Converting to and from string copies the data.
  • The bytes package mirrors strings for byte slices; compare slices with bytes.Equal, not ==.
  • bytes.Buffer grows as you write and shrinks as you read; its zero value is ready to use.
  • io.Writer and io.Reader are one-method interfaces implemented by files, buffers, sockets and more, so the same code works with all of them.
  • strings.NewReader and bytes.NewReader turn in-memory data into readers; io.ReadAll, io.Copy and io.LimitReader are the main helpers.
  • Read returns up to len(p) bytes and io.EOF at the end; process the n bytes before checking the error.
  • bufio batches many small reads and writes. Always Flush a bufio.Writer.

Next: Pointers begins the next chapter, with how Go lets you refer to a value without copying it.