Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 147 · Strings in Depth

Unicode and Multibyte Strings

Handle Unicode text correctly in PHP - bytes vs characters, how UTF-8 works, mb_strlen, mb_substr, mb_strtoupper, mb_str_pad (8.3), mb_trim (8.4) and grapheme pitfalls.

Your users don't all write in English. They're called Zoë, José and Björk; they live in Köln and 東京; they type emoji in comments. PHP handles all of this text perfectly well, as long as you understand one fact: a PHP string is a sequence of bytes, not characters. The classic string functions like strlen, substr and strtoupper count and cut bytes. With non-English text, one character often takes two, three or four bytes, and those functions give wrong lengths, chopped letters and broken output.

The fix is the mbstring extension, whose mb_* functions understand characters. This lesson explains how UTF-8 stores text, shows exactly where byte functions break, and walks through the multibyte versions you need, including the newer mb_str_pad (8.3), mb_trim and mb_ucfirst (8.4). It finishes with the limit even mb_* functions have: user-perceived characters (graphemes) such as flags and family emoji.

You should know the byte-based functions from String Functions first.

Bytes, characters and UTF-8

Unicode gives every character a number called a code point: A is U+0041, é is U+00E9, € is U+20AC, 🙂 is U+1F642. UTF-8 is the way those numbers are stored as bytes, and it's the encoding of the web: almost every web page, database and API uses it.

UTF-8 uses a variable number of bytes per character:

Characters Bytes in UTF-8 Example
ASCII (English letters, digits, punctuation) 1 A = 41
Most Latin accents, Greek, Cyrillic, Arabic, Hebrew 2 é = c3 a9
Most other scripts (Chinese, Japanese, Hindi), € 3 日 = e6 97 a5
Emoji and rare characters 4 🙂 = f0 9f 99 82

Because ASCII characters keep their single byte, plain English text is identical in ASCII and UTF-8. That's why byte functions seem to work fine until the first accented name arrives. You can see the bytes with bin2hex():

PHP Bytes vs characters

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The output (the padding of %-6s also counts bytes, so the columns don't line up for non-ASCII words, which is itself a demonstration):

TEXT
cat    strlen=3  mb_strlen=3  bytes=636174
café  strlen=5  mb_strlen=4  bytes=636166c3a9
naïve strlen=6  mb_strlen=5  bytes=6e61c3af7665
日本 strlen=6  mb_strlen=2  bytes=e697a5e69cac
🙂   strlen=4  mb_strlen=1  bytes=f09f9982
UTF-8

strlen('日本') is 6: two characters, three bytes each. mb_strlen() counts characters. The last line shows the default encoding the mb_* functions assume, UTF-8 (the default since PHP 5.6). You can pass an encoding as the last argument to any mb_* function, but in a modern UTF-8 application you never need to.

Where byte functions break

Here's what goes wrong when byte functions meet UTF-8 text:

PHP What byte functions do to UTF-8 text

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The output:

TEXT
bool(false)
bool(false)
bool(false)
STRAßE IN KöLN
élodie
12

Every result is a bug waiting to happen:

  1. A 4-letter name fails a "maximum 4 characters" check, because it's 5 bytes.
  2. substr('José', 0, 4) keeps Jos plus the first byte of é, producing invalid UTF-8. Browsers show it as a replacement character (�), databases may reject it, and json_encode fails on it.
  3. strrev reverses the two bytes of é into an invalid sequence.
  4. strtoupper and ucfirst only change ASCII letters since PHP 8.2, so ß, ö and é are left alone.
  5. strpos reports byte position 12, but Köln starts at character 10. Feeding a byte position to mb_substr (or a character position to substr) lands in the wrong place.

Byte functions are still correct for many jobs: comparing strings with ===, str_contains, str_replace and explode with UTF-8 search strings, and anything on ASCII-only data like URLs, codes, hashes and HTML tags. They go wrong when you count, cut, pad, reverse or change the case of human text.

The core mb_ functions

Most byte functions have a multibyte twin with the same parameters, counting characters instead of bytes:

Byte function Multibyte version
strlen mb_strlen
substr mb_substr
strpos, stripos, strrpos mb_strpos, mb_stripos, mb_strrpos
substr_count mb_substr_count
strtoupper, strtolower mb_strtoupper, mb_strtolower
ucfirst, lcfirst mb_ucfirst, mb_lcfirst (8.4)
ucwords mb_convert_case($s, MB_CASE_TITLE)
str_split mb_str_split (7.4)
str_pad mb_str_pad (8.3)
trim, ltrim, rtrim mb_trim, mb_ltrim, mb_rtrim (8.4)
ord, chr mb_ord, mb_chr (7.2)
PHP Counting, cutting and searching characters

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The output is 14, Grüße, Köln, 10, aus, the array of four letters J, o, s, é, then ésoJ and Ünïcödé tex…. Every position and length is in characters, so the results match what a human sees.

Changing case correctly

mb_strtoupper and mb_strtolower use Unicode case rules, including special cases where one letter becomes two:

PHP Unicode-aware case conversion

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The output:

TEXT
STRASSE IN KÖLN
ça va, élodie?
Élodie Dupont-Ñúñez
Élan vital
éCOLE
bool(false)
bool(true)

German ß has no single-letter uppercase form in standard case mapping, so mb_strtoupper turns it into SS, which means the uppercase string has more characters than the original. Don't assume case conversion keeps the length the same. MB_CASE_TITLE capitalizes the first letter of every word, including after the hyphen. For locale-specific rules (Turkish dotted and dotless i, for example), the right tool is the intl extension, which isn't installed on our server.

Padding and trimming (PHP 8.3 and 8.4)

str_pad counts bytes, so columns of international text come out ragged. mb_str_pad (8.3) counts characters. Similarly, trim only removes ASCII whitespace, while mb_trim (8.4) also strips Unicode spaces such as the no-break space (U+00A0) and the ideographic space (U+3000) that often sneak in from copied text:

PHP mb_str_pad and mb_trim

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

In the first table, the rows with ë, ö and í are one space short, because each of those letters is two bytes. The mb_str_pad table lines up. Then trim leaves the input untouched (string(12): 2 + 3 + 5 + 2 bytes), while mb_trim returns string(5) "hello".

Even mb_str_pad can't align columns for every script: Chinese and Japanese characters and most emoji are displayed twice as wide as Latin letters in a terminal. mb_strwidth() returns the display width (mb_strwidth('日本') is 4), and mb_strimwidth() truncates to a display width.

Validating and converting encodings

Text from outside your application (uploads, old databases, other systems' CSV files) isn't always valid UTF-8. A common culprit is Latin-1 (ISO-8859-1) or Windows-1252, where é is the single byte e9. Check and convert at the boundary, right when data enters your program:

PHP Checking and converting encodings

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The output starts with bool(false), then café and bool(true). mb_scrub swaps the invalid byte for ?, giving string(5) "caf?!". json_encode returns bool(false) with the message Malformed UTF-8 characters, possibly incorrectly encoded. Finally, € is code point 8364 = U+20AC, and mb_chr(0x1F642) prints 🙂.

htmlspecialchars() also checks UTF-8: since PHP 8.1 it replaces invalid sequences with � by default (the ENT_SUBSTITUTE flag) instead of returning an empty string as older versions did. utf8_encode()/utf8_decode() did the Latin-1 conversion in old code, but they're deprecated since PHP 8.2; use mb_convert_encoding().

Regular expressions need the u flag

PCRE patterns also work on bytes unless you add the u modifier. With u, the pattern and subject are treated as UTF-8, . matches a whole character, and invalid UTF-8 makes the match fail instead of producing garbage:

PHP
preg_match('/^\w+$/', 'café');   // 0: without u, é is two non-word bytes
preg_match('/^\w+$/u', 'café');  // 1
preg_split('//u', 'añb', -1, PREG_SPLIT_NO_EMPTY); // ['a', 'ñ', 'b']

Always add u when matching human text. Regular Expressions covers the flags in detail.

Graphemes: when one "character" is several code points

The mb_* functions count code points. Usually one code point is one visible character, but not always. Some things a reader sees as a single character are built from several code points:

PHP When code points are not characters

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The output (the %-22s padding counts bytes, so the first label's column is one space short):

TEXT
precomposed é         bytes= 2  code points=1
e + combining accent   bytes= 3  code points=2
flag (two letters)     bytes= 8  code points=2
family emoji           bytes=18  code points=5
thumbs up, skin tone   bytes= 8  code points=2
bool(false)

Each of these displays as one symbol, called a grapheme (or grapheme cluster), yet mb_strlen reports up to 5. Consequences:

  • mb_substr can still split a family emoji or a flag, leaving fragments.
  • Two strings that look identical (é precomposed vs e plus a combining accent) aren't === equal. Unicode normalization (Normalizer::normalize()) converts them to one form.
  • A "max 20 characters" limit counts differently from what users see.

The full solution lives in the intl extension: grapheme_strlen(), grapheme_substr(), Normalizer and locale-aware collation. It's widely installed on real servers but not available in our editor, so the code below is read-only:

PHP
// Requires the intl extension
echo grapheme_strlen("\u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F467}"); // 1
echo grapheme_substr('🇯🇵🇫🇷🇧🇷', 0, 2);                                  // 🇯🇵🇫🇷
$normalized = Normalizer::normalize("e\u{0301}", Normalizer::FORM_C);   // "é" (1 code point)

For most applications, mb_* functions are enough: they never produce invalid UTF-8, and they handle every Latin, Cyrillic, Greek, Arabic and CJK letter correctly. Reach for intl when you truncate text containing emoji, compare user-typed names, or sort by language rules.

A UTF-8 checklist

  • Store and transmit text as UTF-8 everywhere: files, HTTP (Content-Type: text/html; charset=UTF-8), and databases (utf8mb4 in MySQL, not the 3-byte utf8).
  • Validate or convert at the boundary with mb_check_encoding() and mb_convert_encoding().
  • Use mb_strlen, mb_substr, mb_strtoupper, mb_str_pad and mb_trim for human text; byte functions are fine for ASCII data and exact matching.
  • Add the u flag to regular expressions on user text.
  • Escape output with htmlspecialchars($s, ENT_QUOTES, 'UTF-8') (see Preventing XSS).
  • Remember that mb_* counts code points, not graphemes; use intl where emoji and combining marks matter.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What does strlen('naïve') return in a UTF-8 file?

    The ï takes two bytes in UTF-8, so the five characters use six bytes. mb_strlen returns 5.
  2. 2. Why is substr('José', 0, 4) dangerous?

    substr counts bytes. The fourth byte is the first half of é, so the result ends with an incomplete character. mb_substr counts characters instead.
  3. 3. What does mb_strtoupper('straße') return?

    Standard Unicode case mapping turns ß into SS, so the result is one character longer. strtoupper() would leave ß unchanged.
  4. 4. Which function trims a no-break space (U+00A0) from the ends of a string?

    trim() only removes ASCII whitespace by default. mb_trim() also removes Unicode whitespace like U+00A0 and U+3000.
  5. 5. A family emoji 👨‍👩‍👧 displays as one symbol. What does mb_strlen report for it?

    mb_* functions count code points. The family emoji is three people joined by two zero-width joiners. grapheme_strlen() from intl would report 1.

Summary

  • PHP strings are bytes. UTF-8 stores ASCII in 1 byte and other characters in 2 to 4 bytes, so byte functions miscount and can cut characters in half.
  • The mbstring functions count characters (code points): mb_strlen, mb_substr, mb_strpos, mb_str_split, mb_strtoupper/mb_strtolower, mb_convert_case.
  • Newer additions: mb_str_pad (8.3), mb_trim/mb_ltrim/mb_rtrim and mb_ucfirst/mb_lcfirst (8.4).
  • Case conversion can change length (ß becomes SS); mb_strwidth measures display width for CJK and emoji.
  • Check and convert input with mb_check_encoding, mb_convert_encoding and mb_scrub; use the u flag in regular expressions.
  • mb_* functions don't understand graphemes (flags, family emoji, combining accents). The intl extension (grapheme_*, Normalizer) does; it's not available in our editor.

Next: Formatting Strings