Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 171 · Strings

Locales and Unicode Text

Understand how C++ handles text beyond ASCII. Use std::locale and facets for number formatting, work with UTF-8 bytes and code points, decode and validate text, and avoid common pitfalls.

A std::string in C++ is a sequence of bytes. That is perfect for English letters, but the world writes in many more scripts: accented letters, Cyrillic, Arabic, Chinese, emoji. Two separate questions come up as soon as your program leaves ASCII. First, how are numbers, dates and sorting rules displayed for a particular culture? C++ answers with locales. Second, how are characters from every script encoded into bytes? The modern answer is Unicode, stored as UTF-8.

This lesson covers both, because they are often confused. You will use std::locale and facets to change how numbers print, see what UTF-8 does to size(), decode and validate UTF-8 by hand, and learn which standard functions to avoid. It builds on Strings, Characters and Booleans and String Formatting.

Locales: culture-specific rules

A locale is an object that bundles cultural rules: the decimal separator (3.14 or 3,14), digit grouping (1,234,567 or 1.234.567), currency, date order, and how to compare and classify characters. Each rule lives in a facet. The facets you will meet first:

Facet Controls
std::numpunct<char> Decimal point, thousands separator, grouping, true/false names
std::ctype<char> Character classes and case conversion (isalpha, toupper)
std::collate<char> String ordering for sorting
std::num_put / std::num_get Converting numbers to and from text
std::money_put, std::time_put Money and date/time text

There are two special locales. The classic ("C") locale is plain: . as decimal point, no digit grouping, ASCII rules. It is what every program starts with. The global locale is the one new streams copy when they are created; std::locale::global(loc) changes it. A locale built from the empty name, std::locale(""), asks the operating system for the user's preferred settings.

flowchart TD
    L["std::locale"] --> F1["numpunct<br/>(decimal point, grouping)"]
    L --> F2["ctype<br/>(character classes, case)"]
    L --> F3["collate<br/>(sort order)"]
    L --> F4["time_put, money_put, ..."]
    S["std::ostream"] -- "imbue(loc)" --> L

A stream owns a locale; the locale is a set of facets that decide how values are written.

Changing how numbers print

The named locales available ("de_DE.UTF-8", "en_US.UTF-8") depend on the machine, and constructing one that does not exist throws std::runtime_error. That makes them poor for examples you want to run everywhere. You do not need an installed locale, though. You can build your own by deriving from a facet and overriding its do_... functions:

C++ A custom numpunct facet

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with std::cin or std::getline), it gets the next line from this box.

Four points to take from this program. A stream gets its locale with imbue, and that affects only that stream. std::format and std::println use the classic locale by default, which makes them predictable; add L to a replacement field ({:L}) when you want locale-aware output, and pass a locale as the first argument to std::format to choose which. And a missing named locale throws, so wrap such code in a try block (Exceptions) or fall back to std::locale::classic().

A common production bug: a program calls std::locale::global(std::locale("")) at startup, and suddenly a number written with << prints 3,14 and the parser reading the same file on another machine fails. When writing data meant for a file, a network or another program, keep the classic locale, or use std::to_chars/std::from_chars, which ignore locales entirely (see Converting Between Strings and Numbers).

Sorting and case

std::collate compares strings according to cultural rules, so an accented letter sorts next to its base letter. A locale object is also a valid comparator: std::locale("de_DE.UTF-8")(a, b) returns true when a sorts before b. This works for installed locales only, and byte-wise < on UTF-8 strings sorts by code point value, which is rarely what a human expects ("Z" < "a" < "é"). For exact multilingual sorting, case folding and normalization, C++ has no complete answer; production code uses a library such as ICU.

Unicode and UTF-8

Unicode assigns every character a number called a code point, written U+0041 for A, U+00E9 for é, U+65E5 for 日 and U+1F642 for the slightly smiling face. UTF-8 is a way to store code points as bytes:

Code point range Bytes Bit pattern
U+0000 to U+007F 1 0xxxxxxx
U+0080 to U+07FF 2 110xxxxx 10xxxxxx
U+0800 to U+FFFF 3 1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Plain ASCII text is identical in UTF-8, which is one reason UTF-8 won. Any byte that starts with 10 is a continuation byte; the others start a new character. Because of this design you can find character boundaries without decoding everything.

The consequence for C++: std::string::size() counts bytes, not characters. "é" has size 2, "日" has size 3, and an emoji has size 4. Indexing with s[i] returns a single byte, which may be the middle of a character. Operations such as substr, reverse and find on raw bytes can cut a character in half and produce invalid text.

Modern compilers read UTF-8 source files and store narrow string literals as UTF-8 (GCC and Clang do this by default; MSVC needs /utf-8). So the practical advice from Characters and Booleans holds: store text as UTF-8 in std::string, and handle code points explicitly when you must.

Counting, decoding and validating

The next program decodes UTF-8 into code points, reverses text correctly, and rejects invalid bytes:

C++ Working with UTF-8 code points

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with std::cin or std::getline), it gets the next line from this box.

A few details worth knowing. The 0xC0 mask test (next & 0xC0) != 0x80 is how a decoder spots a continuation byte. substr(0, 4) of "café 日本 🙂" returns the three letters of caf plus only the first byte of é, which is not valid UTF-8; if you display it, the terminal shows a replacement character. A production decoder must also reject overlong encodings (a short character stored in too many bytes), values above U+10FFFF and the surrogate range U+D800 to U+DFFF. The program above is a teaching decoder, not a security-grade one. In real projects use a tested library for validation, especially for text that arrives from the network.

The other character types

C++ also has types for fixed-width encodings:

Type Literal Holds
char8_t / std::u8string u8"..." UTF-8 code units (C++20)
char16_t / std::u16string u"..." UTF-16 code units (surrogate pairs for emoji)
char32_t / std::u32string U"..." One full code point per element
wchar_t / std::wstring L"..." 16 bits on Windows (UTF-16), 32 bits on Linux and macOS

A std::u32string makes character-by-character processing easy, because each element is a whole code point. It uses four bytes per character and still does not equal what a person sees as a "character": é can be one code point (U+00E9) or two (e followed by a combining accent U+0301). A user-perceived character is called a grapheme cluster. Counting, cursor movement and truncating text for display need grapheme-aware code (ICU, or the Unicode algorithms in a library).

The conversion facets in <codecvt> and std::wstring_convert were deprecated in C++17 and are being removed, so do not build new code on them. Convert with the operating system (MultiByteToWideChar on Windows), with a small function such as the one above, or with a Unicode library. The standard also gained std::print, which outputs UTF-8 correctly on Windows consoles when the source is UTF-8, which is one more reason to prefer it over std::cout for text (Output).

Functions to be careful with

  • std::toupper, std::isalpha, std::islower (from <cctype>) work on single bytes and the C locale only. They do nothing useful for é stored as two bytes. Cast the argument to unsigned char first for ASCII text; passing a negative char is undefined behavior (Undefined Behavior).
  • std::towupper and friends work on wchar_t using the C locale, so results for non-ASCII letters depend on the platform.
  • std::string::length() is the byte count; std::string_view behaves the same (String View).
  • std::setw and {:10} width count code units or code points, not display columns. East Asian characters are usually two columns wide and combining marks have zero width, so tables with non-ASCII text may misalign. The standard formatter estimates width by code points, and a few ranges (CJK, emoji) count as 2.
  • Comparison with == is byte-wise. Two strings that look identical may differ if one uses é as one code point and the other as e plus an accent. Normalize both to the same form before comparing.

A good rule: treat text as opaque UTF-8 bytes whenever you can (store, copy, concatenate, compare for equality, search for ASCII delimiters). Decode only when a feature really needs code points, and bring in a Unicode library for case, collation, normalization and line breaking.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What does std::string("日本").size() return when the source is UTF-8?

    Each of these characters takes three bytes in UTF-8, and size() counts bytes. There are two code points but six bytes.
  2. 2. Which statement about std::format("{}", 1234567) and std::format("{:L}", 1234567) is true?

    Plain replacement fields are locale-independent; adding L uses the locale (the global one, or the one passed to std::format), so digit grouping can appear.
  3. 3. How can a UTF-8 decoder recognize a continuation byte?

    Continuation bytes have the bit pattern 10xxxxxx, so (byte & 0xC0) == 0x80. Lead bytes start with 0, 110, 1110 or 11110.
  4. 4. Why is calling std::toupper on each byte of a UTF-8 string not a correct way to uppercase "é"?

    The two bytes of é are not letters on their own, so toupper leaves them unchanged. Real Unicode case mapping needs a Unicode library.
  5. 5. What happens when you construct std::locale("xx_XX.UTF-8") for a locale the system does not have?

    Named locales depend on the machine. Catch the exception and fall back to std::locale::classic(), or avoid named locales for portable behavior.

Summary

  • A locale is a set of facets that decide how numbers, money, dates, case and sort order work for a culture. Streams get one with imbue.
  • New programs start in the classic ("C") locale; std::format and std::println are locale-independent unless you use {:L}.
  • Named locales depend on the machine and throw when missing. Build your own facets (for example, a numpunct) when you need a predictable format.
  • Use classic formatting or to_chars/from_chars for data that other programs read.
  • Unicode gives each character a code point; UTF-8 stores it in 1 to 4 bytes. std::string::size() counts bytes, and slicing or reversing bytes can corrupt text.
  • Store text as UTF-8 in std::string; decode to code points (or std::u32string) only when needed, and validate untrusted input.
  • A user-perceived character can span several code points; use a Unicode library for normalization, collation, case folding and grapheme handling.

Next: If and Else, where your programs start making decisions.