Skip to content
Tutorials Center
My learning

Topics

Go Reference · bytes, unicode and text

Go unicode/utf8 package

Overview of Go's unicode/utf8 package: count runes, decode and encode UTF-8, validate bytes, handle RuneError and understand why len differs from rune count.

Go source files and strings are UTF-8, in which one character takes between one and four bytes. The unicode/utf8 package works at that byte level: it counts runes, decodes the rune at a position, encodes a rune into bytes, and checks whether data is valid UTF-8.

len("héllo") is 6 because é takes two bytes. utf8.RuneCountInString("héllo") is 5, the number of characters. The package is how you move correctly between those two views.

Go
s := "héllo"
fmt.Println(len(s), utf8.RuneCountInString(s))

Try it

Go Bytes versus runes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Go Encoding, validation and errors

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

What the package contains

Function Purpose
RuneCountInString, RuneCount Number of runes in a string or []byte.
DecodeRuneInString, DecodeRune First rune and its size in bytes.
DecodeLastRuneInString, DecodeLastRune Last rune and its size.
EncodeRune, AppendRune Write a rune as UTF-8 into a slice.
RuneLen Bytes needed for a rune (1 to 4), or -1 if it can't be encoded.
ValidString, Valid, ValidRune Validity checks.
FullRune, FullRuneInString, RuneStart Help when reading partial data in chunks.
Constants RuneError (U+FFFD), UTFMax (4), RuneSelf (0x80), MaxRune

Usage notes

  • Invalid input is not an error. DecodeRune on a bad byte returns (RuneError, 1), and on empty input (RuneError, 0). The size tells you whether it was a real U+FFFD (size 3) or a decoding failure (size 1).
  • range does the decoding for you. for i, r := range s produces the byte offset and rune, and substitutes RuneError for bad bytes. Use utf8 explicitly when you need random access or partial buffers.
  • Rune count is O(n). It scans the whole string; cache it if you need it repeatedly.
  • Slicing by byte index can cut a character. s[:2] of "é!" is half of é. Check utf8.RuneStart(s[i]) or convert to []rune when cutting by character count.
  • Surrogates are invalid. ValidRune(0xD800) is false and RuneLen returns -1 for them.
  • Cleaning input: strings.ToValidUTF8(s, "?") replaces invalid sequences.
  • Grapheme clusters are not runes. An emoji with a skin tone or a letter plus combining accent is several runes; counting user-perceived characters needs golang.org/x/text.