Skip to content
Tutorials Center
My learning

Topics

C# Reference Β· Strings and Text

Unicode normalization in C#

Reference for C# string.Normalize() and IsNormalized(): NFC, NFD, NFKC and NFKD, composed and decomposed text, comparing accents, and removing diacritics.

Unicode can write the same visible text in more than one way. The letter Γ© can be one character (U+00E9) or an e followed by a combining acute accent (U+0065 U+0301). The two look identical on screen but are different strings: they have different lengths and == says they are not equal.

Normalization converts text to a standard form so that equivalent text becomes identical. In .NET you use string.Normalize() and string.IsNormalized(), with a NormalizationForm. Do it before comparing, hashing or storing text from users or files, when equivalence matters.

C#
using System.Text;

string composed = "Γ©";
string decomposed = "é";
bool equal = composed == decomposed;                                       // false
bool afterNormalize = composed == decomposed.Normalize(NormalizationForm.FormC); // true on a normal machine

Try it

C# Two spellings of the same letter

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

C# Forms and ASCII behavior

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Syntax

C#
public string Normalize()
public string Normalize(NormalizationForm normalizationForm)
public bool IsNormalized()
public bool IsNormalized(NormalizationForm normalizationForm)

public enum NormalizationForm { FormC = 1, FormD = 2, FormKC = 5, FormKD = 6 }
Form Meaning
FormC Canonical composition: combine base letter and marks into single characters where possible. The usual choice for storage and comparison.
FormD Canonical decomposition: split into base letters and combining marks. Useful for stripping accents.
FormKC Compatibility composition: also replaces look-alike characters (such as the ligature fi or full-width letters) with plain equivalents.
FormKD Compatibility decomposition.

Returns: a new normalized string (or the same instance if it already was). IsNormalized returns a bool.

Errors / undefined behavior: ArgumentException if the string contains an invalid code point sequence such as a lone surrogate. Behavior depends on the platform's normalization data: with full ICU or NLS data, normalization works as the Unicode standard says; in invariant globalization mode, non-ASCII text is returned as it is, IsNormalized reports false for it, and the invalid-sequence check is skipped.

Usage notes

  • Equality of accented text. "Γ©" == "é" is false. Normalize both sides to the same form (usually FormC) before comparing, or compare with a culture-sensitive comparison that treats canonical equivalents as equal. Ordinal comparison never does.
  • Where to normalize. Normalize at the boundary: when reading user input, file names from different operating systems (macOS tends to produce decomposed names), and before storing keys or usernames.
  • Compatibility forms lose information. FormKC/FormKD turn "β‘ " (circled one) into "1", which is good for search keys and bad for text you must reproduce.
  • Removing accents is done by decomposing, dropping non-spacing marks, and composing again, as in the example. It does not handle every letter: ΓΈ (o with stroke) has no decomposition.
  • Not case folding. Normalization does not change case. Combine with ToUpperInvariant() for case-insensitive keys; see ToUpper() and ToLower().
  • Not a security filter on its own. Normalize before validating, not after, or an attacker can slip in characters that become dangerous after normalization.
  • Length changes. After FormD a string can be longer; after FormC shorter. Do not cache lengths across the call.
  • Runner limit. Our server runs with invariant globalization, where Normalize() is effectively a no-op for non-ASCII text. Read the examples above as showing the API; run them locally with a normal .NET install to see the conversions.