Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 182 · Strings

Unicode, Rune and Encoding

Understand UTF-16 code units, surrogate pairs, Rune, text elements, Encoding.UTF8 byte counts and Unicode normalization in C#, with runnable examples and common pitfalls.

Text looks simple until you meet an emoji, an accented letter typed two different ways, or a file that arrives as raw bytes. Then three questions appear: how many "characters" does this string have, how many bytes will it take on disk, and why does "é" == "é" sometimes return false? All three answers come from one idea: C# keeps what a character is (Unicode) separate from how it is stored (an encoding).

This lesson builds the vocabulary: code points, UTF-16 code units, Rune, text elements (what a reader would call a character) and bytes. You will count each of them for the same string, convert text to and from UTF-8, and learn what normalization does. It builds on bool and char and Strings.

Four ways to count "characters"

Unicode gives every character a number called a code point, written U+0041 for A or U+1F600 for the grinning-face emoji. There are about 1.1 million possible code points. In memory, a C# string is a sequence of 16-bit UTF-16 code units (char values). A code point up to U+FFFF fits in one code unit. A larger one is stored as a surrogate pair: a high surrogate followed by a low surrogate, two char values that only mean something together.

So string.Length counts UTF-16 code units, not what a person sees:

C# Length counts UTF-16 code units

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The two lone surrogates cannot be encoded, so UTF-8 replaces each with the three bytes EF BF BD, the encoding of U+FFFD, the replacement character �. The last line shows exactly that: EFBFBD twice, followed by 206968, the bytes of " ih". Any code that walks a string char by char (reversing, truncating, splitting at a fixed index) can cut a pair in half. The rest of the lesson shows the tools that don't.

A lone surrogate is a legal string in C# but not legal Unicode text. It can slip through Substring, Remove or Truncate-style helpers, and it fails later (at an encoder, a JSON writer, a database) rather than where you made the mistake.

Rune: a whole code point

System.Text.Rune is a struct that holds exactly one valid Unicode code point (any value from U+0000 to U+10FFFF except the surrogate range). Where char is a storage unit, Rune is a character in the Unicode sense. A Rune can't be a lone surrogate: the constructor throws and Rune.TryCreate returns false.

string.EnumerateRunes() walks a string one code point at a time, combining surrogate pairs for you. A lone surrogate comes out as the replacement character instead of corrupting your loop.

C# Walking a string with EnumerateRunes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Each row of that output tells you something. The euro sign is one UTF-16 unit but three UTF-8 bytes; the emoji is two units and four bytes. Those two counts, Utf16SequenceLength and Utf8SequenceLength, are the reason Length and "byte size" are different questions.

When should you reach for Rune? When you must classify or transform individual code points (is this a letter, a digit, a symbol?) and want emoji or rare scripts to work. char.IsLetter(char) can't answer for half of a surrogate pair, but Rune.IsLetter can. For everyday ASCII-style parsing, plain char is fine and faster to write.

Text elements: what readers call a character

Even a Rune isn't always one thing on screen. Unicode builds some visible symbols from several code points:

  • A base letter plus a combining mark: e followed by U+0301 (combining acute accent) displays as é.
  • Emoji joined by the zero-width joiner (U+200D): man + ZWJ + woman + ZWJ + girl is drawn as one family glyph.
  • A base emoji plus a skin-tone modifier, and flags made of two "regional indicator" letters.

The user-visible unit is called a grapheme cluster, and .NET exposes it as a text element through System.Globalization.StringInfo.

C# Chars, runes, text elements

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Use text elements whenever a number must match what a user sees: a "max 20 characters" username field, cursor movement in an editor, truncating a preview with an ellipsis. For counting 'visible characters' this is the closest .NET gets; the exact rules follow the Unicode text-segmentation standard and improve between .NET releases. Rendering is a different matter, since fonts decide whether a family emoji draws as one glyph or three people side by side.

From text to bytes: encodings

A string lives in memory as UTF-16, but files, sockets and databases deal in bytes. An encoding is the rule that maps text to bytes and back. System.Text.Encoding has one static property per common encoding:

Property Bytes per code point Notes
Encoding.UTF8 1 to 4 The default on the web and in most files. ASCII text stays 1 byte per character.
Encoding.Unicode 2 or 4 UTF-16 little endian, the in-memory form of a string.
Encoding.BigEndianUnicode 2 or 4 UTF-16 big endian.
Encoding.UTF32 4 Always four bytes, so it is wasteful.
Encoding.Latin1 1 Only the first 256 code points; everything else becomes ?.
Encoding.ASCII 1 Only 0 to 127; everything else becomes ?.
C# The same text in six encodings

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Read the UTF-8 line carefully: 41 C3A9 E282AC F09F9880. Each code point takes the shortest form that fits (1, 2, 3 and 4 bytes), so the byte count of a UTF-8 string is neither Length nor the number of runes. GetByteCount gives you the exact size before you allocate, and GetMaxByteCount the safe upper bound. The "..."u8 literal, covered in Raw, Verbatim and UTF-8 String Literals, skips the conversion at runtime.

Notice what the lossy encodings did. Latin1 kept é (it is code point 233) but replaced the euro sign and the emoji's two UTF-16 units with ?, so three question marks. ASCII lost the é too. After that, the damage is permanent: converting back gives Aé???, not your original text.

Bad bytes and split sequences

Decoding is where real data bites. Bytes you receive may not be valid in the encoding you assume. By default .NET never throws: invalid bytes become U+FFFD (�). That is safe for display but silently destroys data, so strict decoders exist for formats where corruption must be caught.

A second trap is chunking. When you read a file or network stream in blocks, a multi-byte character may be split across two reads. Decoding each block separately with GetString breaks it; a stateful Decoder remembers the incomplete bytes between calls.

C# Invalid input and split characters

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The mojibake line (café) is the classic symptom of reading UTF-8 bytes as Latin1: the two bytes of é become two separate characters. If you see à followed by an odd symbol in user data, some layer assumed the wrong encoding.

Two more practical notes:

  • The BOM. A byte order mark (EF BB BF in UTF-8) is an optional signature at the start of a file. Encoding.UTF8 reports a 3-byte preamble, so writers that use it (such as StreamWriter(path, false, Encoding.UTF8)) prefix files with a BOM, while File.WriteAllText(path, text) defaults to UTF-8 without one. If a tool complains about stray characters at the start of a file, check for a BOM. Reading with File.ReadAllText detects it and skips it for you.
  • Other code pages. Legacy encodings such as Windows-1252 are not built in on .NET Core and newer; they need Encoding.RegisterProvider(CodePagesEncodingProvider.Instance). In this sandbox Encoding.GetEncoding("windows-1252") throws ArgumentException because the provider is not registered, which is the same thing you would see locally without that line. Prefer UTF-8 for anything new.

Normalization: one text, several spellings

Because Unicode allows é both as one code point (U+00E9, precomposed) and as e plus U+0301 (decomposed), two strings that look identical can differ in their code units. Ordinal comparison, dictionary keys and hashing all see them as different.

Normalization converts text to a canonical form so equal-looking strings become equal. The four forms are:

Form What it does
FormC Compose: use precomposed characters wherever possible. The best default for storing and comparing text.
FormD Decompose: split into base letters plus combining marks. Useful for stripping accents.
FormKC Compose, and also replace "compatibility" characters (the fi ligature becomes fi, ① becomes 1).
FormKD Decompose with compatibility replacements.

On a normal machine, the code below prints False for the plain comparison and True after Normalize(). The sandbox here runs .NET in invariant globalization mode, which has no Unicode normalization data, so Normalize() returns text unchanged. This example detects the difference and tells you which world it ran in:

C# Composed and decomposed text

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

On a machine with normal globalization data, the same program prints that Normalize(FormC) made them equal. Typical rule of thumb: normalize external text to FormC once, at the boundary where it enters your system (a form field, an import file), then compare with StringComparison.Ordinal afterwards. Don't normalize on every comparison, and never normalize passwords before hashing unless the protocol says to.

Case conversion has Unicode surprises too. "ß".ToUpperInvariant() stays "ß" because .NET maps one char to one char and has no single-character capital for it, while the Turkish dotted İ (U+0130) and dotless ı break the assumption that "I".ToLower() is "i" in every culture. Prefer StringComparison.OrdinalIgnoreCase for identifiers and protocols, as covered in Strings.

Choosing the right unit

Question Use
How many UTF-16 units / array size for a char[]? string.Length
How many Unicode code points? EnumerateRunes().Count()
How many visible characters? StringInfo.LengthInTextElements
How many bytes in a file or network message? Encoding.UTF8.GetByteCount
Is this symbol a letter, digit, emoji? Rune.GetUnicodeCategory, Rune.IsLetter
Do these two texts mean the same thing? Normalize to FormC, then compare

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What does "hi \U0001F600".Length return?

    The emoji is outside the 16-bit range, so it occupies a surrogate pair (two chars). Three chars for "hi " plus two for the emoji makes five.
  2. 2. Which type represents exactly one valid Unicode code point and cannot hold half of a surrogate pair?

    Rune is a struct for a single scalar value. Its constructor throws for surrogate values, unlike char which can hold either half of a pair.
  3. 3. A family emoji is built from several code points joined by zero-width joiners. Which API counts it as ONE visible character?

    Text elements follow grapheme-cluster rules, so the whole joined sequence is one element. Length counts UTF-16 units and EnumerateRunes counts code points.
  4. 4. You read a UTF-8 file as Latin1 and see café instead of café. What happened?

    The two UTF-8 bytes of é were each turned into a separate Latin1 character. Decode with the encoding the data was written in.
  5. 5. Why should you use a stateful Decoder when reading a UTF-8 stream in blocks?

    Decoding each block on its own would turn the halves of a split character into replacement characters. A Decoder carries leftover bytes into the next call.

Summary

  • A C# string is UTF-16. Length counts code units; characters above U+FFFF use a surrogate pair (two chars).
  • Rune is a full Unicode code point. Use EnumerateRunes() and Rune methods to handle emoji and rare scripts correctly.
  • A visible character may be several code points (combining marks, ZWJ sequences, flags). StringInfo counts and walks these text elements.
  • Encoding.UTF8 turns text into 1 to 4 bytes per code point; GetByteCount gives the exact size. Lossy encodings (ASCII, Latin1) replace what they can't represent with ?.
  • Invalid bytes decode to � unless you ask for strict decoding. Use a Decoder for chunked input and decode with the encoding the data was written in.
  • Normalize external text to FormC before comparing text that must be equal when it looks equal (needs ICU/globalization data at runtime).

Next: Regular Expressions