Skip to content
Tutorials Center
My learning

Topics

Lesson 88 of 147 Β· Standard Library

Unicode and Text Processing

Handle Unicode text in Go - the unicode and utf8 packages, invalid UTF-8, combining marks, bufio.Scanner splitters, text/scanner and strings.NewReplacer.

The Strings, Bytes and Runes lesson explained the core model: a Go string is a sequence of bytes, usually UTF-8, and a rune is one Unicode code point. That is enough for English text and for a lot of international text. But real-world text contains accented letters written in two different ways, digits that are not 0-9, emoji that take two UTF-16 units, bytes that are not valid UTF-8 at all, and characters that are two columns wide on a terminal.

This lesson is the toolbox for those cases. You will learn the unicode package for classifying characters, unicode/utf8 and unicode/utf16 for encoding details, what to do with invalid input, why "same-looking" strings can compare as different, how to split text with bufio.Scanner splitters and text/scanner, and how strings.NewReplacer replaces many patterns in one pass. At the end you will also see where the standard library stops and golang.org/x/text begins.

Classifying characters with the unicode package

The unicode package answers "what kind of character is this?" for any rune. The helpers are named after the Unicode general categories: letters (L), marks (M), numbers (N), punctuation (P), symbols (S) and separators (Z).

Go Asking Unicode what a rune is

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Several things in that output are worth slowing down for:

  • IsLetter('δΈ–') is true, but neither IsUpper nor IsLower is: many scripts have no case.
  • The Arabic-Indic digit Ω£ (U+0663) is both IsDigit and IsNumber. unicode.IsDigit is not the same as "between '0' and '9'". If you want ASCII digits only, compare the rune yourself: r >= '0' && r <= '9'.
  • IsNumber is wider than IsDigit: it also accepts things such as Β½ and β…¦.
  • The character Η… (U+01C5) is a titlecase letter, which is neither upper nor lower. The three-way split between ToUpper, ToLower and ToTitle exists for such letters.
  • IsSpace covers space, tab, newline and other Unicode spaces. IsPunct is true for _ and !, but symbols such as + or $ are IsSymbol, not punctuation.
  • unicode.Is(table, r) tests membership in a script (unicode.Han, unicode.Greek, unicode.Latin) or a category (unicode.Letter, unicode.Mn). In takes several tables and IsOneOf takes a slice.

When you validate identifiers or usernames, spell the rule out. "Letters, digits and underscore" in unicode terms (IsLetter || IsDigit || r == '_') allows Cyrillic and Arabic letters, which may or may not be what you want. If only ASCII is acceptable, write that as an explicit range check instead of relying on IsLetter.

The unicode/utf8 package

unicode/utf8 works at the encoding level: how runes turn into bytes and back. You met RuneCountInString already. These are the functions you will reach for next:

Go Decoding, validating and encoding UTF-8

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The functions fall into a few groups:

  • Decode: DecodeRuneInString(s) returns the first rune and how many bytes it used. DecodeLastRuneInString does the same from the end, which is how you walk a string backwards. The []byte versions are DecodeRune and DecodeLastRune.
  • Validate: ValidString and Valid check a whole string or slice; ValidRune checks a single value (surrogate halves such as 0xD800 are not valid runes).
  • Encode and measure: RuneLen is how many bytes a rune needs (or -1 if it cannot be encoded), and AppendRune appends the encoded form to a byte slice without allocating a string.
  • Boundaries: RuneStart(b) is true if a byte begins a rune (continuation bytes look like 10xxxxxx), and FullRuneInString tells you whether the data holds a complete encoded rune. Network code uses these when a rune may be cut in half between two reads.

Invalid UTF-8

A Go string can hold any bytes, including ones that are not valid UTF-8: data from files, network peers, legacy encodings, and users. Go does not stop you. Instead, decoding replaces each bad byte with the replacement character U+FFFD (utf8.RuneError, displayed as οΏ½) and moves forward by exactly one byte. You can see this in the output above: range over "a\xffb\xc3" yields four runes, two of them U+FFFD.

DecodeRuneInString returns (RuneError, 1) for bad input. Be careful: a valid, genuine U+FFFD in the text decodes with a size of 3, so the size tells you whether the decoding failed (r == utf8.RuneError && size == 1).

Use strings.ToValidUTF8(s, replacement) to clean a string before you store it or send it to something that insists on valid UTF-8 (JSON encoders, databases, XML). Replacing silently loses information, so for strict inputs check ValidString first and reject the request.

UTF-16 and surrogate pairs

Go strings are UTF-8, but other systems are not: JavaScript and Java strings, Windows APIs and the JSON \uXXXX escape all work in UTF-16, where characters outside the Basic Multilingual Plane (such as most emoji) need two 16-bit units called a surrogate pair. unicode/utf16 converts between the two worlds.

Go UTF-8 versus UTF-16 lengths

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The same three characters take 8 bytes in UTF-8, 4 units in UTF-16 and are 3 runes. That mismatch is behind a classic bug: a JavaScript client that reports "text length 4" for Aβ‚¬πŸ™‚ while your Go server counts 3 runes. When two systems must agree on offsets, decide up front which unit you count in. For appending one rune to a []uint16 without a temporary slice, utf16.AppendRune exists, mirroring utf8.AppendRune.

When equal-looking strings differ

Unicode allows many characters to be written in more than one way. The letter Γ© can be one code point (U+00E9, precomposed) or an e followed by a combining acute accent (U+0301, decomposed). Both display as Γ©, and a person cannot tell them apart. Go compares strings byte by byte, so it can:

Go Two ways to write cafΓ©

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The two strings print identically and are even the same "length" to a human, but == is false, len differs (5 and 6) and the rune counts differ (4 and 5). Several consequences follow:

  • Equality, map keys and database lookups treat them as different values. A user who types cafΓ© on one keyboard and pastes it on another may produce different bytes.
  • Reversing runes breaks combining marks. The reversed decomposed string puts the accent first, where it attaches to nothing (or to the previous character). A "character" as people see it, called a grapheme cluster, can be several runes.
  • Case rules are not symmetric. strings.EqualFold uses simple case folding, so "Straße" is not equal to "STRASSE" (that needs full folding, where ß becomes ss), while Ξ£ and Ο‚ and even the Kelvin sign K (U+212A) do fold to the same letter. ToUpper("straße") leaves ß alone because the single-rune uppercase form does not exist in simple mappings.

The fix for the first problem is normalization: converting text to a canonical form (NFC composes, NFD decomposes) before you compare or store it. The standard library does not include a normalizer. The official extended library does:

Go
import "golang.org/x/text/unicode/norm"

a := norm.NFC.String("café") // "café" with U+00E9
b := norm.NFC.String("cafΓ©")
fmt.Println(a == b)                 // true

The same module provides golang.org/x/text/cases (full case mapping and title-casing), golang.org/x/text/collate (language-aware sorting, because sorting by code point is wrong for most languages), golang.org/x/text/width (East Asian width) and golang.org/x/text/encoding (converting from Latin-1, Shift-JIS and other legacy encodings). Those are read-only here because the sandbox only has the standard library, but they are the right tools in real projects.

[]rune conversions and range are right for most business text, but not for user-visible "characters". If you must cut, count or reverse text as a human sees it (for example, truncating a username at 20 characters), use a grapheme-cluster segmenter such as github.com/rivo/uniseg, not []rune.

Display width

A terminal or a fixed-width table cares about columns, which are neither bytes nor runes. Combining marks take no column, and East Asian wide characters take two. fmt pads by runes, so %-8s misaligns CJK text. This small function estimates width for a few common cases:

Go Bytes, runes and columns

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

ζ—₯本θͺž has 9 bytes and 3 runes but occupies 6 columns, so %-8s pads it to eight runes and the closing bracket lands too far right. The estimate above is deliberately small; real width tables (emoji, ambiguous-width characters, wide punctuation) are what x/text/width or github.com/mattn/go-runewidth supply. The last lines show strings.Map, a handy way to filter runes: returning -1 drops the character, here removing control characters such as NUL and tab.

Splitting text: bufio.Scanner and split functions

bufio.Scanner reads a stream piece by piece, and a split function decides what a "piece" is. The standard library ships four: ScanLines (the default), ScanWords (runs separated by Unicode white space), ScanRunes (one UTF-8 rune at a time, with invalid bytes returned as U+FFFD) and ScanBytes.

Go Built-in and custom split functions

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Notice that ScanWords keeps punctuation attached (fun. and Unicode:), because "word" means "run of non-space characters". The ScanRunes token for πŸ™‚ is one whole 4-byte rune, never half of one.

A custom SplitFunc receives the unread bytes data and a flag atEOF. It returns how many bytes to consume (advance), the token to hand to Scan, and an error. The contract is:

  • To deliver a token, return advance > 0 and the token bytes.
  • To ask for more data, return 0, nil, nil. The scanner then reads more and calls you again with the bigger buffer. Our function does this at the end of the last word when atEOF is false: it cannot know the word is finished until it sees a non-letter or the end of input.
  • Return advance greater than zero with a nil token to skip input (our function skips the non-letters in front of a word by consuming them together with the first token).
  • Use utf8.DecodeRune to step over whole runes. Looping over bytes and testing each one would cut Γ― or Γ© apart.

Because the scanner buffers tokens, there is a limit: a single token larger than 64 KiB makes Scan fail with bufio.ErrTooLong. Call sc.Buffer(make([]byte, 0, 64*1024), 1<<20) before scanning to allow longer lines, and always check sc.Err() after the loop. The Reading Input lesson shows this in context.

Tokenizing source-like text with text/scanner

text/scanner is a small lexer for text that looks like Go source: identifiers, numbers, strings, characters and comments. It is useful for simple configuration formats, calculators and mini languages. It reports each token's position and skips white space and comments by default.

Go Tokens from text/scanner

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Scan returns the next token as a rune: one of the constants scanner.Ident, Int, Float, Char, String, RawString, Comment and EOF, or, for any other character, the character itself (so + is returned as the rune '+'). TokenText gives the matching text. Two behaviours to expect: the scanner does not know that := or >= are single operators, so they arrive as two tokens each (combine them in your parser), and the // add tax comment never appears because comments are skipped by default. Set s.Mode to control that, and s.IsIdentRune to change what an identifier may contain (for example to allow -).

For anything with a real grammar, such as full programming languages, you will write a proper lexer, as in the Markdown converter project. For a handful of token types, text/scanner saves the work.

Replacing many things at once with strings.NewReplacer

Chained strings.ReplaceAll calls have a hidden danger: later calls see the output of earlier ones. To swap a and b you would replace a with b, and then every b (including the new ones) with a, and end up with no as at all. strings.NewReplacer does all replacements in one pass over the input and never re-scans its own output.

Go One-pass replacements

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The rules of matching are simple: at each position the replacer tries the old strings in the order you listed them, and the first one that matches wins. That is why "a" before "aa" turns "aaa" into 111 (the "aa" rule never gets a chance), while listing the longer pattern first gives 21. Put longer patterns before their prefixes.

Replace(s) returns a new string; WriteString(w, s) writes the result straight to an io.Writer and reports the number of bytes written, which avoids building an intermediate string. A replacer is safe for concurrent use, so create it once (a package-level variable works well) and reuse it.

Common uses are HTML escaping (although html.EscapeString already exists, and html/template escapes automatically), converting typographic quotes and dashes to ASCII for legacy systems, and cleaning up identifiers. For patterns, not fixed strings, reach for regular expressions.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. What does utf8.RuneCountInString("a\xffb") return?

    Each byte that is not part of a valid UTF-8 sequence is counted as one rune (U+FFFD, utf8.RuneError). The string has a, one bad byte and b.
  2. 2. Given a := "café" and b := "café", what does a == b evaluate to?

    Go compares strings byte by byte. The first uses the precomposed Γ© (2 bytes); the second uses e plus a combining accent (3 bytes). Normalize both with golang.org/x/text/unicode/norm before comparing.
  3. 3. Why is unicode.IsDigit(r) not a safe test for "ASCII digit"?

    IsDigit is true for any character in the Unicode category Nd, including digits from other writing systems. For ASCII only, test r >= '0' && r <= '9'.
  4. 4. What does strings.NewReplacer("a", "1", "aa", "2").Replace("aaa") return?

    At each position the first matching old string in argument order wins. "a" is listed before "aa", so every a becomes 1. List "aa" first to get 21.
  5. 5. What should a bufio.SplitFunc return when the data holds only part of a token and atEOF is false?

    Returning 0, nil, nil tells the scanner to read more input and call the function again with a larger buffer.

Summary

  • The unicode package classifies runes by category and script (IsLetter, IsDigit, IsSpace, Is(unicode.Han, r)). Its results are broader than ASCII, so write explicit ranges when you need ASCII only.
  • unicode/utf8 decodes, validates, measures and appends runes. Invalid bytes decode as U+FFFD with size 1; clean strings with strings.ToValidUTF8 or reject them with ValidString.
  • unicode/utf16 converts to and from UTF-16, where emoji use surrogate pairs, so lengths differ between UTF-8, UTF-16 and runes.
  • Equal-looking strings can differ in bytes (composed versus decomposed accents). Normalization, full case folding, collation and width live in golang.org/x/text, not in the standard library.
  • A rune is not always a user-perceived character; combining marks and emoji sequences need grapheme-aware libraries.
  • bufio.Scanner split functions (ScanLines, ScanWords, ScanRunes, or your own) decide what a token is; text/scanner tokenizes Go-like text.
  • strings.NewReplacer performs many replacements in a single pass, with earlier arguments taking priority.

Next: Packages and Visibility