Python Reference · Standard Library: Math, Data and Text
The unicodedata module
Reference for Python's unicodedata module: look up character names and categories, normalize text with NFC and NFD, strip accents and compare strings safely.
The unicodedata module gives access to the Unicode Character Database: the official names, categories, numeric values and decomposition rules for every character Python can hold in a str. Its most important function is normalize(), which solves the problem that the same visible text can be stored as different sequences of code points.
For example, é can be one code point (U+00E9) or an e followed by a combining accent (U+0065 U+0301). They look identical but compare unequal until you normalize both to the same form.
import unicodedata print(unicodedata.name("é")) # LATIN SMALL LETTER E WITH ACUTE
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input · what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input · what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Main functions
| Function | Returns |
|---|---|
name(chr, default) |
The official name, such as "LATIN SMALL LETTER A". Raises ValueError if there is no name and no default. |
lookup(name) |
The character with that name. Raises KeyError if unknown. |
category(chr) |
A two-letter category code: Lu (uppercase letter), Ll, Nd (decimal digit), Zs (space separator), Sc (currency symbol), Po (punctuation) and so on. |
normalize(form, text) |
The text in form "NFC", "NFD", "NFKC" or "NFKD". |
is_normalized(form, text) |
True if already in that form (Python 3.8+). |
combining(chr) |
The combining class; non-zero for combining marks such as accents. |
decimal(chr), digit(chr), numeric(chr) |
The character's decimal, digit or numeric value (numeric("½") is 0.5). |
bidirectional(chr), east_asian_width(chr), mirrored(chr) |
Text direction class, display width class and mirroring flag. |
decomposition(chr) |
The decomposition mapping as a string. |
unidata_version |
The Unicode version the module was built with. |
Normalization forms
| Form | Meaning |
|---|---|
NFC |
Compose: combine base letters and marks into single code points where possible. The usual choice for storage and comparison. |
NFD |
Decompose: split precomposed characters into base plus marks. |
NFKC |
Compose, and also replace compatibility characters (ligatures like fi, full-width digits, superscripts) with plain equivalents. |
NFKD |
Decompose with compatibility replacements. |
Usage notes
- Normalize before comparing or hashing user text, such as usernames, file names or dictionary keys. Use NFC for equality, and NFKC when visually confusable variants should be treated alike.
name()takes exactly one character. A longer string raisesTypeError.- Stripping accents works by NFD plus removing characters whose
combining()is non-zero. It does not map letters likeø,łorß. - Case-insensitive comparison is
str.casefold(), notlower(). Combine it with normalization for best results. - Category checks beat ranges.
category(c).startswith("L")identifies any letter, in any script;str.isalpha()is built on the same data. - Python versions bring newer Unicode data. Newly added characters may have no
name()on older interpreters. - Code points are not graphemes. A single "character" the user sees (like an emoji with a skin tone) can be several code points;
len()counts code points.
Related
str.casefold(),str.isalpha(),str.encode(),ord(),chr()- Lessons: Strings