Skip to content
Tutorials Center
My learning

Topics

Lesson 82 of 135 ยท Standard Library

Locales and Wide Characters

Handle international text in C - set the locale with setlocale, use wchar_t and wide strings, convert UTF-8 with mbstowcs, and avoid byte-versus-character bugs.

A C char is one byte, and for English text one byte is one letter. The moment your program meets รฉ, รŸ, โ‚ฌ or ๆ—ฅๆœฌ่ชž, that rule breaks. Under UTF-8 (the encoding almost every modern system uses) the euro sign takes three bytes, an emoji takes four, and strlen happily reports bytes, not characters. Add the fact that some countries write 3,14 where others write 3.14, and you have the two problems this lesson solves: locales, which describe local conventions, and wide characters, which let you work with one element per character.

You will learn what setlocale changes, how to read the current conventions, how wchar_t and the wcs* functions work, how to convert between UTF-8 bytes and wide strings, and the fixed-width types char16_t and char32_t from <uchar.h>. You should already be comfortable with strings, characters and the string functions.

What a locale is

A locale is a bundle of local rules that the C library consults while it runs. It is split into categories, each controlled separately:

Category What it affects
LC_CTYPE Which bytes are letters, upper case, printable; the multibyte encoding (UTF-8 or not)
LC_COLLATE The sort order used by strcoll
LC_NUMERIC The decimal point (and digit grouping) in printf, strtod
LC_MONETARY Currency symbol and format (read through localeconv)
LC_TIME Month and day names, date formats for strftime
LC_ALL All of the above at once

Every program starts in the minimal "C" locale, regardless of your computer's language settings. It is deliberately boring: ASCII only, a . as the decimal point, and plain byte comparison. That is why a freshly started program cannot print รฉ correctly through wide functions or classify it with isalpha; nobody has told it your language yet.

You select a locale with setlocale, from <locale.h>:

C
char *setlocale(int category, const char *name);
  • setlocale(LC_ALL, "") means "use whatever the user's environment says" (the LANG and LC_* environment variables). This is what most programs call once at the start of main.
  • setlocale(LC_ALL, "C.UTF-8") or "en_US.UTF-8" asks for a named locale.
  • setlocale(LC_ALL, NULL) changes nothing and returns the current name.

It returns the new locale name, or NULL if the locale is not installed on the machine. Always check for NULL, because a locale that exists on your laptop may not exist on a server:

C Asking for locales and checking the result

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

The start-up locale is always C, and the made-up locale fails with NULL while the previous setting stays. localeconv() returns a struct lconv that describes the numeric and monetary conventions of the active locale: decimal_point, thousands_sep, currency_symbol and more. The locales that exist depend on the machine, and the names are not portable: Linux and macOS use en_US.UTF-8; Windows uses names such as English_United States.1252 or .UTF8.

A warning for programs that read numbers: after setlocale(LC_ALL, "") on a German system, printf("%.2f", 3.14159) prints 3,14 and strtod("3.14", NULL) stops at the dot. If your program writes data files or talks to other programs, either keep LC_NUMERIC in the "C" locale (call setlocale(LC_NUMERIC, "C") after the first call) or use the locale only for text. See string conversions for the number parsers.

Bytes are not characters

Before reaching for wide characters, look at what a UTF-8 string really is. Each character (a Unicode code point) is encoded in 1 to 4 bytes:

Code point range Bytes Example
U+0000 to U+007F 1 A
U+0080 to U+07FF 2 รฉ
U+0800 to U+FFFF 3 โ‚ฌ
U+10000 to U+10FFFF 4 ๐Ÿ˜€

The first byte tells you how long the sequence is, and every following byte (a continuation byte) looks like binary 10xxxxxx. That makes UTF-8 easy to take apart even without any library. The next example decodes a string by hand, so you can see exactly what strlen is counting. It needs no locale at all:

C Decoding UTF-8 by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

Four characters, ten bytes. The "quick count" trick (count every byte that is not a continuation byte) is the simplest correct way to get the length of a valid UTF-8 string, and it works whatever the locale is. It is also the reason slicing a UTF-8 string at a random byte index can cut a character in half, so truncate at code point boundaries.

For many tasks you never need more than this: copying, searching with strstr, splitting on ASCII separators and printing all work on UTF-8 bytes safely, because no ASCII byte value ever appears inside a multi-byte sequence. You only need characters when you count, index, change case or compare text by language rules.

wchar_t and wide strings

For those jobs C has wide characters. The type wchar_t (from <wchar.h>) holds one character, and a wide string is a null-terminated array of wchar_t. Wide literals start with L:

C
wchar_t letter = L'รฉ';
const wchar_t *word = L"hรฉllo";

Each narrow function has a wide twin that works on wchar_t, so you can index by character:

Narrow Wide Header
strlen, strcpy, strcmp, strstr wcslen, wcscpy, wcscmp, wcsstr <wchar.h>
printf, sprintf, snprintf wprintf, swprintf (always bounded) <wchar.h>
isalpha, isupper, toupper iswalpha, iswupper, towupper <wctype.h>
strtol, strtod wcstol, wcstod <wchar.h>

In wprintf, a %ls conversion prints a wide string and %lc a wide character; %s expects a narrow string. How wide is wchar_t? The standard leaves that open. On Linux and macOS it is 32 bits and holds any Unicode code point. On Windows it is 16 bits (UTF-16), so a character outside the first 65,536 code points needs two wchar_t units. Do not store wchar_t data in files or send it over a network, because its size and byte order vary; convert to UTF-8 first.

C Wide strings and wide classification

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

Notice that this program never calls printf. That is on purpose, as the next section explains. Notice too that towupper and iswalpha only know about รฉ and รถ because setlocale selected a UTF-8 locale. In the plain "C" locale the classification of non-ASCII characters is not guaranteed.

Converting between UTF-8 and wide strings

Files, terminals, sockets and source code are all bytes, so a real program usually reads UTF-8, converts to wide strings when it needs per-character work, and converts back to write. The conversion functions depend on the locale's multibyte encoding, so call setlocale with a UTF-8 locale first:

flowchart LR
    A["UTF-8 bytes<br/>char[]"] -->|"mbstowcs / mbrtowc"| B["wide string<br/>wchar_t[]"]
    B -->|"wcstombs / wcrtomb"| A
    B --> C["per-character work:<br/>wcslen, towupper, iswalpha"]

Convert at the edges, work in wide characters in the middle.

  • mbstowcs(dest, src, max) converts a whole multibyte string. Pass NULL as dest to get just the number of characters, which is the cleanest way to count them.
  • wcstombs(dest, src, max) goes the other way; with NULL it returns the number of bytes needed.
  • Both return (size_t)-1 if the input is invalid in the current encoding.
  • mbrtowc and wcrtomb convert one character at a time and keep their state in an mbstate_t, which is what you want when decoding a stream chunk by chunk.
C mbstowcs, wcstombs and error handling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

The output mixes narrow printf with these conversions. That is fine here, because we never call a wide output function on the same stream. Here the upper-cased copy has the same size as the original, but in general case mapping can change length (German รŸ becomes SS under full Unicode rules), which per-character towupper cannot express. For real linguistic text processing, use a library such as ICU.

Stream orientation

Here is a rule that surprises many people. Each FILE * stream, including stdout, is unoriented until its first use. The first call decides: a byte function (printf, puts, fputs) makes it byte-oriented, and a wide function (wprintf, putwchar) makes it wide-oriented. After that, the other kind of call does not work. It is not an error you will see; the output simply fails and returns a negative value.

stateDiagram-v2
    [*] --> Unoriented
    Unoriented --> Byte: printf, puts, fputs
    Unoriented --> Wide: wprintf, putwchar
    Byte --> Byte: printf, puts
    Wide --> Wide: wprintf, putwchar

Once a stream is byte-oriented, wide output calls fail, and the other way round. Choose one style per stream.

C Mixing printf and wprintf

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

Line 2 never appears. The practical advice: do not mix. The simplest robust style, used by most Unix programs, is to keep all I/O as narrow UTF-8 bytes and convert to wchar_t in memory only when you need to. Then printf("%s", utf8) just copies the bytes, and the terminal draws them.

Sorting by language rules

strcmp compares bytes. In ASCII every capital letter sorts before every lower-case one, so "Banana" comes before "apple". strcoll compares using LC_COLLATE and gives the order a reader expects:

C strcmp versus strcoll

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

Passing strcoll to qsort (see sorting and searching) sorts a word list in dictionary order. Collation rules are language-specific (Swedish sorts รค after z, German treats it like a), which is exactly why a locale is a parameter and not a constant.

Fixed-width character types

Because wchar_t varies by platform, C11 added <uchar.h> with two types whose encoding is fixed:

  • char16_t holds a UTF-16 code unit; literals look like u"text". Characters beyond U+FFFF use two units (a surrogate pair).
  • char32_t holds a full code point (UTF-32); literals look like U"text".
  • u8"text" is a UTF-8 string literal. Since C23 its elements have type unsigned char.

mbrtoc32 decodes the next code point from a multibyte string, and c32rtomb encodes one. The type is a good fit when you want a guaranteed-32-bit code point on every platform, including Windows. As with the wide functions, set a UTF-8 locale first.

C
#include <locale.h>
#include <uchar.h>
#include <string.h>

setlocale(LC_ALL, "C.UTF-8");
char32_t code_point;
mbstate_t state = {0};
size_t length = mbrtoc32(&code_point, "โ‚ฌ", 3, &state);   /* 3, code_point == 0x20AC */

Compile-time constants help with sizes: sizeof(u"๐Ÿ˜€") / sizeof(char16_t) is 3, two units for the emoji plus the terminator, while U"๐Ÿ˜€" takes two elements, one code point plus the terminator.

Practical guidance

  • Treat text as UTF-8 bytes everywhere you store or transmit it. Convert to wchar_t/char32_t only inside the functions that need per-character work.
  • Call setlocale(LC_ALL, "") once at program start if you want the user's language settings, and always check for NULL when you request a specific locale.
  • Keep numeric parsing and number output locale-independent in file formats and protocols.
  • Never mix printf and wprintf on the same stream.
  • Do not assume sizeof(wchar_t); do not write raw wchar_t arrays to disk.
  • For real internationalization (normalization, case folding, sorting, line breaking), use a library like ICU. The standard C library only handles the basics.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. On a UTF-8 system, strlen("nรฉโ‚ฌ") returns what?

    strlen counts bytes up to the terminator: n is 1 byte, รฉ is 2 and โ‚ฌ is 3. The locale does not change that.
  2. 2. Which call counts the characters of a UTF-8 string in a UTF-8 locale?

    mbstowcs with a NULL destination returns how many wide characters the conversion would produce. wcslen needs a wide string, not a char string.
  3. 3. What does setlocale(LC_ALL, "") do?

    The empty string means "use the environment's settings". Reset with "C", and query with NULL.
  4. 4. A program calls printf first and later wprintf on stdout. What happens to the wprintf output?

    The first I/O call fixes a stream's orientation. A wide call on a byte-oriented stream does nothing and returns a negative value.
  5. 5. Why should you not write raw wchar_t arrays to a file you plan to share between platforms?

    wchar_t is 4 bytes on Linux and macOS and 2 bytes on Windows, with different byte orders possible. Store UTF-8 instead.

Summary

  • A locale holds local conventions in categories (LC_CTYPE, LC_COLLATE, LC_NUMERIC, LC_TIME, LC_MONETARY); programs start in the minimal "C" locale until you call setlocale.
  • setlocale(LC_ALL, "") adopts the user's settings, a name picks one explicitly and NULL queries; a NULL result means the locale is not installed.
  • UTF-8 uses 1 to 4 bytes per character, so strlen counts bytes; count code points by skipping continuation bytes or with mbstowcs(NULL, s, 0).
  • wchar_t strings (L"...", wcslen, wprintf, towupper, iswalpha) give one element per character, but the type is 16-bit on Windows and 32-bit elsewhere.
  • Convert at the edges with mbstowcs/wcstombs (or mbrtowc/wcrtomb for streams); a result of (size_t)-1 means invalid input.
  • Never mix narrow and wide output on one stream; strcoll sorts by language rules; char16_t and char32_t have a fixed size.

Next: Complex Numbers with complex.h