Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 122 · Text Content

Character Encoding and UTF-8

Understand bytes vs characters, ASCII, Windows-1252, Unicode and UTF-8, where to put meta charset, and how to diagnose and fix garbled text known as mojibake.

Have you ever seen a web page where "café" turned into café, or where apostrophes became ’? That garbage has a name — mojibake (Japanese for "character transformation") — and it is always caused by the same mistake: a file was saved with one character encoding and read with another.

The good news is that the fix is simple and the same for every new page: use UTF-8 everywhere, and say so. This lesson explains what an encoding is, how we got to UTF-8, the one line of HTML you need, and how to diagnose the problem when it happens anyway.

Bytes are not characters

Computers store files as bytes: numbers from 0 to 255. A text file is just a long list of bytes. To display it, a program needs a rulebook that says which bytes stand for which characters. That rulebook is a character encoding.

Encoding happens in two directions:

  • When you save a file, your editor encodes characters into bytes.
  • When a browser loads the file, it decodes the bytes back into characters.

If both sides use the same rulebook, everything works. If they use different ones, the bytes are the same but the characters come out wrong. Nothing is "lost" yet — the browser is simply reading correct bytes with the wrong rules.

A short history: from ASCII to Unicode

ASCII (1963) defined 128 characters: the English alphabet, digits, punctuation and some control codes. Each fits in 7 bits, so every ASCII character is one byte with a value below 128. ASCII has no é, no ü, no €, and nothing at all from non-Latin scripts.

To fill the remaining byte values from 128 to 255, the world created dozens of 8-bit encodings, each for a different region:

  • ISO-8859-1 (Latin-1) added Western European letters such as é, ñ and ß.
  • Windows-1252 is Microsoft's variant of Latin-1 that also includes curly quotes, the em dash and the euro sign. Browsers actually treat any page labeled ISO-8859-1 as Windows-1252.
  • Other encodings covered Cyrillic, Greek, Arabic, Hebrew, Japanese, Chinese and more — each using the same byte values for different characters.

That last point is the root of mojibake. Byte 233 is é in Windows-1252, й in Windows-1251 (Cyrillic), and something else again elsewhere. A file only made sense if you knew which encoding it used, and you could not mix scripts in one document.

Unicode solved this by giving every character in every writing system a single number, its code point (you met these in Symbols and Emojis). Unicode now covers more than 150,000 characters, from Latin and Chinese to mathematical symbols and emoji.

UTF-8: Unicode as bytes

Unicode lists the characters, but files still need bytes. UTF-8 is the encoding that turns code points into bytes, and it has won: it is used by roughly 98% of all websites, and the HTML standard requires it for new documents.

UTF-8 uses between one and four bytes per character:

Characters Bytes Example
ASCII (English letters, digits, basic punctuation) 1 A → 41
Most other alphabets: accented Latin, Greek, Cyrillic, Arabic, Hebrew 2 é → C3 A9
Most remaining characters, including Chinese, Japanese, Korean and € 3 € → E2 82 AC
Emoji and rarer characters 4 😀 → F0 9F 98 80

Two clever properties made it the favorite. First, every ASCII file is already valid UTF-8, so old English text kept working. Second, multi-byte sequences use byte values that never appear in plain ASCII, so a program can never confuse part of an é with a < or a ".

You can see the bytes for yourself. JavaScript's TextEncoder always produces UTF-8:

Characters to UTF-8 bytes

Add your own characters to the list — your name, a word in another language, a favorite emoji — and see how many bytes each needs.

Declaring the encoding: <meta charset>

A browser cannot reliably guess an encoding from bytes alone, so you tell it. In HTML, that is one line in the <head>:

HTML
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Café Menu</title>
</head>

The rules for this line are short but important:

  • Put it first in the <head>. The declaration must be within the first 1,024 bytes of the file, and nothing that contains text — especially the <title> — should come before it.
  • Use exactly one. A page has only one encoding.
  • utf-8 is the only value you need. It is not case-sensitive; UTF-8 works too.

You may see an older, longer form in legacy pages. It does the same thing; prefer the short one:

HTML
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

The HTTP header

Web servers can also announce the encoding in the HTTP response, before the HTML even arrives:

Text
Content-Type: text/html; charset=utf-8

If the header and the <meta> tag disagree, the header wins. Most modern frameworks, including Laravel, send charset=utf-8 by default, but a misconfigured server can override a perfectly correct <meta> tag. If a page looks wrong even though your HTML is right, check the response headers in the DevTools Network panel (see Viewing Source and Using DevTools).

If neither the header nor the page declares an encoding, the browser falls back to a guess based on the user's language settings — often Windows-1252 for Western languages. That is why an undeclared UTF-8 page can look fine on your computer and broken on someone else's.

About the TryIt editor: previews on this site are loaded straight from text, not from bytes, so a <meta charset> inside a TryIt has no effect there. In real files on a real server, it matters a lot.

Diagnosing mojibake

Mojibake patterns are surprisingly recognizable once you know them. The example below simulates the most common one: text saved as UTF-8, then decoded as Windows-1252.

Mojibake simulator

Every non-ASCII character became two or three odd ones, because each of its UTF-8 bytes was read as a separate Windows-1252 character. Here is a quick guide to what you might see:

Symptom Likely cause Fix
é, ü, ’, — UTF-8 bytes read as Windows-1252 Declare UTF-8 in <meta> and the HTTP header
� (a diamond with a question mark) Legacy bytes read as UTF-8 — they are not valid UTF-8 Re-save the file as UTF-8
??? in place of letters Characters were lost when saved, often in a database Store data as UTF-8 (in MySQL, use utf8mb4) and re-enter it
A stray  at the very top A byte order mark read as Windows-1252 Save as UTF-8 without BOM

The first two are only display problems: the bytes are fine, and the right declaration fixes them. The third means the data itself is damaged, so prevention matters.

The byte order mark (BOM)

Some editors can write three invisible bytes, EF BB BF, at the start of a UTF-8 file. This byte order mark says "this file is UTF-8". Browsers respect it — it even overrides the HTTP header and the <meta> tag.

For HTML you do not need it, and it can cause trouble elsewhere: some server-side languages send it as output before your page begins, and some tools show it as . When your editor offers "UTF-8" and "UTF-8 with BOM", choose plain UTF-8.

A UTF-8 checklist

Encoding problems disappear when every link in the chain agrees:

  1. Editor: save every file as UTF-8 (without BOM). Most modern editors do this by default; check the status bar.
  2. HTML: put <meta charset="utf-8"> first in the <head>.
  3. Server: send Content-Type: text/html; charset=utf-8.
  4. Database: use a UTF-8 character set that supports all of Unicode, including emoji.
  5. Forms: browsers submit form data in the page's encoding, so a UTF-8 page sends UTF-8 data. You will see more in Forms.

URLs have their own way of carrying non-ASCII characters, built on top of UTF-8; that is the topic of URL Encoding.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. A page shows "café" instead of "café". What most likely happened?

    é is two bytes in UTF-8 (C3 A9). Read as Windows-1252, those bytes become à and ©. Declaring UTF-8 fixes it.
  2. 2. Where should <meta charset="utf-8"> go?

    The declaration must appear within the first 1,024 bytes, and nothing with text, such as the title, should come before it.
  3. 3. How many bytes does UTF-8 use for the letter A?

    UTF-8 is compatible with ASCII, so every ASCII character takes exactly one byte.
  4. 4. The server sends charset=iso-8859-1 but your page says <meta charset="utf-8">. Which one does the browser use?

    The HTTP header takes priority over the meta tag (only a byte order mark beats both), so the server configuration must be fixed too.

Summary

  • Files are bytes; a character encoding is the rulebook that maps bytes to characters.
  • ASCII covers 128 characters; 8-bit encodings like Windows-1252 reused the same bytes for different characters, causing mojibake.
  • Unicode numbers every character; UTF-8 stores them in 1–4 bytes and is compatible with ASCII.
  • Put <meta charset="utf-8"> first in the <head>, and make the server send charset=utf-8 — the header wins if they differ.
  • é-style garbage means UTF-8 read as a legacy encoding; � means the opposite.
  • Save files as UTF-8 without BOM, and keep databases and forms in UTF-8 too.

Next: Times, Data Values and Ruby Text — giving dates and numbers a machine-readable form.