Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 55 · Data Representation

Text and Character Encoding

Learn how computers store text: ASCII, Unicode code points, UTF-8 bytes, emoji, and why broken characters (mojibake) appear when a file is read with the wrong encoding.

A computer only stores numbers, so text has to be turned into numbers before it can be saved, sent or shown. The letter "A" is not stored as a picture of an A. It is stored as a number, and every program that touches the text must agree on which number means which letter. That agreement is called a character encoding.

Most of the time you never notice encodings, until they go wrong: a name like "José" turns into "José", a web page fills up with question marks, or an emoji becomes an empty box. This lesson explains how text becomes bytes, why one character can take several bytes, and how to recognize and fix the classic mistakes. It builds on Binary, Octal and Hexadecimal and Bits and Bytes.

From letters to numbers: ASCII

Early computers needed a shared table. The best-known one is ASCII (American Standard Code for Information Interchange), created in the 1960s. It assigns a number from 0 to 127 to each of 128 symbols, so every code fits in 7 bits.

Range What it holds Examples
0-31 and 127 control codes 10 = line feed (new line), 9 = tab
32-47 space and punctuation 32 = space, 33 = !
48-57 digits 48 = 0, 57 = 9
65-90 capital letters 65 = A, 90 = Z
97-122 lowercase letters 97 = a, 122 = z

Notice that the digit character "5" has code 53, not 5. The character and the number it looks like are different things. Also notice that a (97) is exactly 32 more than A (65): the two alphabets differ by a single bit, which made case conversion cheap on old hardware.

ASCII covers English well and almost nothing else. There is no "é", no "ß", no Cyrillic, Arabic, Chinese or emoji. Because a byte has 8 bits, there were 128 spare values, and every region filled them differently. Latin-1 added Western European letters, other tables added Greek or Cyrillic, and the same byte could mean different letters depending on the table. A file sent between countries often came out garbled.

Unicode: one number for every character

Unicode solves the problem by giving every character in every writing system its own number, called a code point. Code points are written as U+ followed by hexadecimal digits:

  • U+0041 is the letter A (the same value as in ASCII, so ASCII is a subset of Unicode).
  • U+00E9 is é.
  • U+20AC is the euro sign €.
  • U+1F600 is the grinning face emoji.

Unicode defines over a million possible code points and has assigned well over 150,000 of them, covering modern and historic scripts, mathematical symbols and emoji. Notice what Unicode is not: it is a numbering of characters, not a way of storing them. It does not say how many bytes a code point takes. That job belongs to an encoding.

Do not mix up the three words. A character set or code space is the list of characters and their numbers (Unicode). A code point is one number in that list. An encoding (UTF-8, UTF-16, UTF-32) turns code points into bytes.

UTF-8: the encoding the web uses

Several encodings of Unicode exist. UTF-32 uses 4 bytes for every code point: simple, but wasteful for English text. UTF-16 uses 2 or 4 bytes and is used inside JavaScript, Java and Windows. UTF-8 uses 1 to 4 bytes per code point, and it is the dominant encoding for files and web pages.

UTF-8 is clever in three ways:

  1. Code points 0-127 use one byte, identical to ASCII. Any pure ASCII file is already valid UTF-8.
  2. Larger code points use 2, 3 or 4 bytes. The first byte tells you how long the sequence is.
  3. Every continuation byte starts with the bits 10, so a program that lands in the middle of a text can find the start of the next character.

The patterns look like this (the x positions hold the bits of the code point):

Code point range Bytes Bit pattern
U+0000 to U+007F 1 0xxxxxxx
U+0080 to U+07FF 2 110xxxxx 10xxxxxx
U+0800 to U+FFFF 3 1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
flowchart LR
  C["Character<br/>€"] --> P["Code point<br/>U+20AC"]
  P --> E["UTF-8 encoder"]
  E --> B["Bytes<br/>E2 82 AC"]
  B --> S["Saved in a file<br/>or sent over a network"]

A character becomes a code point, and the encoding turns the code point into the bytes that are actually stored.

The first TryIt below runs on a real Node.js server. It shows each character's code point and the bytes UTF-8 produces for it, and then writes the encoder rules out by hand and checks them against Node's own encoder. The Buffer object in Node holds raw bytes, which makes it perfect for looking underneath a string.

Node.js Code points and UTF-8 bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Study the second block of output. The euro sign needs three bytes: the leading byte 11100010 starts with 1110, so it announces a three-byte sequence, and the two bytes after it each start with 10.

Characters, code points and bytes are different counts

A common beginner mistake is to assume that "length" means one thing. For a piece of text there are at least three different numbers:

  • the number of bytes it takes in a given encoding,
  • the number of code points (what Unicode counts),
  • the number of user-perceived characters (what a reader would count).

Even these three do not always agree. JavaScript strings are sequences of UTF-16 units, so an emoji outside the first 65,536 code points counts as two units in .length. Some visible characters are built from several code points: the letter "é" can be one code point (U+00E9) or the letter "e" followed by a combining accent (U+0301). They look identical but compare as different strings until you normalize them.

Node.js Three ways to count the same text

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The practical rule: when a limit is given in "characters" (a username field, an SMS), find out whether it means bytes, code points or something else. A database column limited to 255 bytes holds far fewer Chinese characters than English ones.

Mojibake: what happens when the encoding is wrong

Bytes carry no label. A file is just a list of numbers, and a program has to decide how to read them. If the writer used UTF-8 and the reader assumes Latin-1 (or the other way round), every multi-byte character is decoded as several wrong characters. This garbage has a Japanese name, mojibake ("character transformation").

The classic symptoms are easy to recognize:

  • "café" shows up as "café": UTF-8 text read as Latin-1. The two bytes C3 A9 are shown as two separate characters.
  • Black diamonds with question marks, or U+FFFD (the replacement character): bytes that are not valid in the encoding used to read them.
  • "©" and similar doubled sequences: text that was decoded wrongly and then saved again as UTF-8, damaging it twice.

The next TryIt reproduces these mistakes and also shows the repair when the damage happened only once: encode the wrongly decoded text back to the same bytes, then decode those bytes the right way.

Node.js Breaking and repairing text

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The last result is the replacement character, U+FFFD. The byte E9 on its own is not valid UTF-8, so the decoder gives up and inserts a marker. That marker replaces the real information, so a file that has been repaired by replacement cannot be restored.

How to avoid it

  • Use UTF-8 everywhere unless a system forces something else: files, databases, web pages, APIs.
  • Declare the encoding where the format allows it. HTML pages should contain <meta charset="utf-8">, and servers can send Content-Type: text/html; charset=utf-8.
  • Never "fix" mojibake by retyping the text: find the step where the encoding changed (an editor setting, a database connection, an export option).
  • When opening a file, if the text looks broken, try another encoding; most editors let you reopen a file with a chosen one.

Emoji and the wider picture

Emoji are ordinary Unicode characters. The grinning face is U+1F600 and needs four bytes in UTF-8. Many emoji are sequences: a family emoji can be several people joined by an invisible joiner character (U+200D), and skin-tone variants add a modifier code point. One symbol on screen can therefore be many code points and well over ten bytes. Whether you see a picture at all depends on whether your fonts have a glyph for it, which is why older devices show an empty box for new emoji.

Unicode also affects sorting, searching and upper-casing. The German "ß" becomes "SS" in capitals, Turkish has a dotted and a dotless "i", and users expect "Ä" to sort in a sensible place. Languages and databases provide locale-aware tools; plain byte comparison is only right for a few cases.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What is a Unicode code point?

    A code point is the number Unicode assigns to a character. An encoding such as UTF-8 then decides which bytes represent that number.
  2. 2. How many bytes does the euro sign (U+20AC) take in UTF-8?

    Code points from U+0800 to U+FFFF use three bytes in UTF-8. The euro sign is stored as E2 82 AC.
  3. 3. A web page shows "café" instead of "café". What most likely happened?

    The two UTF-8 bytes C3 A9 for "é" were decoded one byte at a time using Latin-1, producing two wrong characters.
  4. 4. Why is any plain ASCII file also a valid UTF-8 file?

    UTF-8 was designed so that code points 0-127 are encoded as one byte with the same value as in ASCII.

Summary

  • Computers store text as numbers; a character encoding is the agreement about which numbers mean which characters.
  • ASCII has 128 codes and covers basic English. Unicode gives every character in every script a unique code point.
  • UTF-8 stores each code point in 1 to 4 bytes and is backward compatible with ASCII; it is the best default for files and the web.
  • Bytes, code points and visible characters are different counts, and equal-looking text can have different code points until normalized.
  • Mojibake appears when bytes are read with a different encoding than the one used to write them; declare and use UTF-8 consistently.

Next, see how pictures, sound and video become numbers too in Images, Sound and Video as Data.