Go Reference · bytes, unicode and text
Go unicode/utf8 package
Overview of Go's unicode/utf8 package: count runes, decode and encode UTF-8, validate bytes, handle RuneError and understand why len differs from rune count.
Go source files and strings are UTF-8, in which one character takes between one and four bytes. The unicode/utf8 package works at that byte level: it counts runes, decodes the rune at a position, encodes a rune into bytes, and checks whether data is valid UTF-8.
len("héllo") is 6 because é takes two bytes. utf8.RuneCountInString("héllo") is 5, the number of characters. The package is how you move correctly between those two views.
s := "héllo" fmt.Println(len(s), utf8.RuneCountInString(s))
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input · what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input · what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.
What the package contains
| Function | Purpose |
|---|---|
RuneCountInString, RuneCount |
Number of runes in a string or []byte. |
DecodeRuneInString, DecodeRune |
First rune and its size in bytes. |
DecodeLastRuneInString, DecodeLastRune |
Last rune and its size. |
EncodeRune, AppendRune |
Write a rune as UTF-8 into a slice. |
RuneLen |
Bytes needed for a rune (1 to 4), or -1 if it can't be encoded. |
ValidString, Valid, ValidRune |
Validity checks. |
FullRune, FullRuneInString, RuneStart |
Help when reading partial data in chunks. |
| Constants | RuneError (U+FFFD), UTFMax (4), RuneSelf (0x80), MaxRune |
Usage notes
- Invalid input is not an error.
DecodeRuneon a bad byte returns(RuneError, 1), and on empty input(RuneError, 0). The size tells you whether it was a real U+FFFD (size 3) or a decoding failure (size 1). rangedoes the decoding for you.for i, r := range sproduces the byte offset and rune, and substitutesRuneErrorfor bad bytes. Useutf8explicitly when you need random access or partial buffers.- Rune count is O(n). It scans the whole string; cache it if you need it repeatedly.
- Slicing by byte index can cut a character.
s[:2]of"é!"is half ofé. Checkutf8.RuneStart(s[i])or convert to[]runewhen cutting by character count. - Surrogates are invalid.
ValidRune(0xD800)is false andRuneLenreturns -1 for them. - Cleaning input:
strings.ToValidUTF8(s, "?")replaces invalid sequences. - Grapheme clusters are not runes. An emoji with a skin tone or a letter plus combining accent is several runes; counting user-perceived characters needs
golang.org/x/text.