Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 67 · Data Representation

Characters, ASCII and Unicode

Learn how computers turn text into bytes: ASCII, Unicode code points, UTF-8 and UTF-16, endianness, byte order marks and the bugs encodings cause.

Memory holds only numbers, so text needs an agreement: which number stands for which character? Such an agreement is a character set. A second agreement says how those numbers are turned into bytes. That is an encoding. Mixing them up, or guessing the wrong one, produces the garbled text known as mojibake.

In this lesson you will follow text from ASCII to Unicode, see exactly how UTF-8 and UTF-16 lay out bytes, learn why byte order matters, and reproduce the classic encoding bugs in Python so you can recognize them later. Python's str is a sequence of Unicode characters and bytes is raw data, which makes it a perfect lab; see also bytes and bytearray.

ASCII: the 7-bit starting point

ASCII (1963) maps 128 characters to the numbers 0 to 127, using 7 bits:

  • 0 to 31 and 127: control characters (newline is 10, tab is 9, NUL is 0).
  • 32: space. 48 to 57: the digits 0 to 9. 65 to 90: A to Z. 97 to 122: a to z.

The layout was designed with tricks in mind. Upper and lower case differ by exactly one bit (value 32, bit 5), so 'A' | 0x20 gives 'a'. A digit's code is 48 + digit, so ord('7') - ord('0') is 7.

ASCII has no e with an accent, no Cyrillic, no emoji. Since computers store bytes (8 bits), many vendors used the spare half (128 to 255) for extra letters, each in a different way: Latin-1 for Western Europe, Windows-1251 for Cyrillic, and dozens more. The same byte meant different characters on different machines. That chaos led to Unicode.

Python ASCII codes and tricks

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Unicode: one number for every character

Unicode is a character set, not an encoding. It assigns a unique number called a code point to each character in every writing system, written U+0041 for A, U+00E9 for é, U+20AC for the euro sign, and U+1F600 for a grinning face. The code space runs from U+0000 to U+10FFFF, which is 1,114,112 possible code points, of which a bit over 150,000 are assigned.

The first 128 code points are identical to ASCII, and the first 256 are identical to Latin-1. A code point is an abstract number; to store it you need an encoding.

One more subtlety: what a reader sees as one "character" may be several code points. é can be a single code point (U+00E9) or e followed by a combining accent (U+0065 U+0301). Flags and many emoji are sequences of code points, too. Text libraries call the user-visible unit a grapheme cluster.

UTF-8: the web's encoding

UTF-8 encodes each code point as 1 to 4 bytes:

Code point range Bytes Bit pattern
U+0000 to U+007F 1 0xxxxxxx
U+0080 to U+07FF 2 110xxxxx 10xxxxxx
U+0800 to U+FFFF 3 1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
flowchart LR
  CP["Code point<br/>U+20AC"] --> BITS["Bits<br/>0010 0000 1010 1100"]
  BITS --> PACK["Pack into<br/>1110xxxx 10xxxxxx 10xxxxxx"]
  PACK --> OUT["Bytes<br/>E2 82 AC"]

The euro sign needs 14 significant bits, so it uses the three-byte pattern.

The design has useful properties:

  • ASCII compatible. Pure ASCII text is already valid UTF-8, byte for byte.
  • Self-synchronizing. A byte starting with 10 is always a continuation byte, so you can find the start of a character from any position.
  • No byte-order problem. The bytes come in one fixed order.
  • No stray zero bytes inside non-NUL characters, so C-style strings still work.

The cost: a character's byte length varies, so len(bytes) is not the number of characters and you cannot index the 5th character in constant time.

Python Encode characters as UTF-8 by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

UTF-16 and UTF-32

UTF-16 uses 16-bit units. Code points up to U+FFFF take one unit (two bytes). Higher ones, such as emoji, take two units called a surrogate pair: subtract 0x10000, put the top 10 bits in a "high surrogate" (0xD800 to 0xDBFF) and the low 10 bits in a "low surrogate" (0xDC00 to 0xDFFF). Windows APIs, Java, JavaScript and .NET strings use UTF-16 internally, which is why "😀".length in JavaScript is 2.

UTF-32 always uses four bytes per code point. It is simple and fast to index but wastes memory, so it is used mostly inside programs.

Encoding A é € 😀
UTF-8 1 byte 2 bytes 3 bytes 4 bytes
UTF-16 2 bytes 2 bytes 2 bytes 4 bytes
UTF-32 4 bytes 4 bytes 4 bytes 4 bytes

Endianness and byte order marks

Any value wider than one byte must be stored in some byte order. Big-endian puts the most significant byte first (at the lowest address), like writing a number normally. Little-endian puts the least significant byte first. x86 and (in practice) ARM systems are little-endian; network protocols use big-endian, called "network byte order".

TEXT
0x12345678 stored at addresses 0, 1, 2, 3
big-endian:    12 34 56 78
little-endian: 78 56 34 12

UTF-16 and UTF-32 have 16- and 32-bit units, so they come in two flavors, UTF-16LE/UTF-16BE. UTF-8 has one-byte units, so endianness does not apply.

A byte order mark (BOM) is the character U+FEFF placed at the start of a file. If a reader sees bytes FF FE it knows the file is little-endian UTF-16; FE FF means big-endian. (The swapped value U+FFFE is guaranteed never to be a character, so there is no ambiguity.) UTF-8 files sometimes start with EF BB BF, the BOM encoded in UTF-8. It carries no byte-order information and is merely a signature. It is unnecessary and often harmful: shell scripts with a BOM fail, and CSV importers show it as an odd first column name. Prefer UTF-8 without a BOM.

Python Endianness and BOMs

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the utf-16 line: Python adds a BOM and uses the machine's native order, while utf-16-le and utf-16-be are explicit and add none.

Encoding bugs and how to spot them

Almost every encoding bug is one of these:

  1. Decoding with the wrong encoding produces mojibake. UTF-8 bytes read as Latin-1 turn é (bytes C3 A9) into é. Seeing é, ’ or  in text is a signature of this bug.
  2. Invalid bytes. Bytes that are not valid in the stated encoding raise an error, or are replaced by U+FFFD (a question mark in a diamond) when decoding with errors="replace".
  3. Cutting strings in the wrong place. Slicing UTF-8 bytes in the middle of a character leaves a broken sequence. Slicing UTF-16 between surrogates splits an emoji.
  4. Lossy encoding. Encoding to ASCII or Latin-1 something that does not fit raises UnicodeEncodeError, or with errors="replace" silently turns it into ?.
  5. Comparing unequal forms. "é" as one code point and as e plus a combining accent look identical but compare unequal until you normalize them (NFC or NFD).
  6. Counting wrongly. Byte length, code point count and visible character count can all differ.
Python Reproduce classic encoding bugs

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rule of thumb: decode bytes to text as soon as data enters your program, work with text inside, and encode back to bytes at the edges. Always say which encoding you mean, and make it UTF-8 unless a format forces something else.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What is the difference between Unicode and UTF-8?

    Unicode is the character set (a number for each character). UTF-8, UTF-16 and UTF-32 are encodings that turn those numbers into bytes.
  2. 2. How many bytes does the euro sign U+20AC take in UTF-8?

    U+20AC lies in the range U+0800 to U+FFFF, which uses the three-byte pattern. The bytes are E2 82 AC.
  3. 3. Why does UTF-8 not need a byte order mark to define its byte order?

    The code unit of UTF-8 is one byte, so there is no multi-byte unit whose order could vary. A BOM in UTF-8 is just a signature.
  4. 4. You see 'café' where 'café' was expected. What most likely happened?

    The UTF-8 bytes C3 A9 read as two separate Latin-1 characters give à and ©, the signature of this mix-up.
  5. 5. In UTF-16, how is a code point above U+FFFF stored?

    Characters outside the Basic Multilingual Plane are split into a high and a low surrogate, which together take four bytes.

Summary

  • A character set maps characters to numbers; an encoding maps those numbers to bytes.
  • ASCII uses 7 bits; Unicode assigns code points up to U+10FFFF and keeps ASCII as its first 128.
  • UTF-8 uses 1 to 4 bytes, is ASCII-compatible and has no byte-order issue; UTF-16 uses 2 or 4 bytes via surrogate pairs; UTF-32 uses 4.
  • Multi-byte units can be big- or little-endian; a BOM (U+FEFF) can signal the order, and a UTF-8 BOM should usually be avoided.
  • Typical bugs are the wrong decoding, invalid bytes, cutting through characters, lossy encoding and unnormalized comparisons.
  • Decode on input, use text internally, encode on output, and prefer UTF-8.

Next, learn how bits survive noisy storage and transmission in Parity, Checksums and Error-Correcting Codes.