Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 133 · Buffers and Binary Data

Text, Encodings and Unicode

Understand Unicode in Node.js - UTF-8 versus UTF-16, string length versus byte length, TextEncoder and TextDecoder, streaming decoding, normalization, Intl.Segmenter, atob/btoa and Buffer base64.

"Text" feels simple until you meet a name like Zoë, an emoji, or a file saved by another program. Then a character turns into ë, a string has a different length than you expect, two identical-looking names refuse to compare equal, and a chat message gets cut in the middle of a flag. All of these come from the same fact: computers store bytes, people read characters, and several layers sit between them.

This lesson untangles those layers. You will learn what Unicode, UTF-8 and UTF-16 are, why '🚀'.length is 2, how TextEncoder and TextDecoder convert between strings and bytes (including in streaming mode and with strict error checking), how normalization makes é equal é, how to count what users see as characters with Intl.Segmenter, and how base64 works with btoa, atob and Buffer. It builds on Buffers and Typed Arrays.

Unicode, code points and encodings

Unicode is a catalogue that gives every character a number called a code point, written U+0041 for A, U+00E9 for é, U+1F680 for the rocket emoji. There are about 1.1 million possible code points, grouped in planes; the first 65,536 (the Basic Multilingual Plane) cover most living languages.

An encoding is a rule for turning code points into bytes:

Encoding Bytes per code point Notes
UTF-8 1 to 4 ASCII stays one byte; the web, files and Node's default
UTF-16 2 or 4 How JavaScript strings are stored in memory; Windows APIs
UTF-32 4 Rare
Latin-1 (ISO-8859-1) 1 Only 256 characters; legacy

Use UTF-8 for everything you store or send unless a format forces otherwise.

Node.js Code points and their UTF-8 bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

UTF-8 uses a clever scheme: the first byte tells how many bytes the character has (0xxxxxxx is one, 110xxxxx two, 1110xxxx three, 11110xxx four) and the continuation bytes always look like 10xxxxxx. That makes it self-synchronising: you can find the start of a character from any position, and invalid data is detectable.

Length: code units, code points, graphemes and bytes

"How long is this text?" has four answers.

Node.js Four ways to measure a string

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

  • UTF-16 code units (string.length): what JavaScript indexes. Characters outside the BMP, such as emoji, use two units (a surrogate pair).
  • Code points ([...string].length, or for...of): one per Unicode character, though one can be an invisible combining mark.
  • Graphemes: what a reader sees as one character. é written as e plus a combining accent is two code points but one grapheme; the family emoji joins three people with two zero-width joiners into one picture: five code points, eight UTF-16 units, one grapheme.
  • Bytes: what you store or send. Size limits (a database column, an HTTP header, a 280-character post) must say which of these they mean.

Cutting a string with slice or substring works on code units, so you can split a surrogate pair in the middle and produce an invalid half character.

Node.js Slicing in the middle of an emoji

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

isWellFormed() and toWellFormed() (both available since Node 20) check for and repair lone surrogates, replacing them with U+FFFD. This matters because a malformed string cannot be encoded as valid UTF-8: encodeURIComponent throws a URIError on one, and JSON.stringify writes it as an escape like \ud83d. Truncate by code points or graphemes, not by slice, when you shorten user text.

TextEncoder: strings to bytes

TextEncoder is a global that always encodes as UTF-8 (the only encoding the standard allows) and returns a Uint8Array.

Node.js TextEncoder basics

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

encodeInto(string, target) writes as many whole characters as fit and reports how many UTF-16 units it read and bytes it written. It never splits a character: here only Zoë fits, because the rocket needs four bytes and just one is left (read: 4, written: 5). Because it avoids allocating a new array, it is useful in hot loops. The last line shows that a lone surrogate becomes the three bytes ef bf bd, the UTF-8 encoding of U+FFFD.

TextDecoder: bytes to strings

new TextDecoder(label, options) decodes bytes to text. The default is UTF-8; many other labels are supported, such as utf-16le, latin1 (an alias for windows-1252 in the web standard) and legacy encodings if Node was built with full ICU (the internationalization library).

Node.js TextDecoder, fatal mode and BOM

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

By default invalid sequences become U+FFFD (�). That is forgiving but can silently corrupt data, so use { fatal: true } when invalid input is an error you want to hear about, for example when validating an upload. The BOM (byte order mark, EF BB BF in UTF-8) is an invisible marker some Windows tools put at the start of a file; TextDecoder strips it by default, whereas buffer.toString() leaves it in as a  character, a classic cause of "my first JSON key does not match".

Buffer.from(bytes).toString('utf8') is the Node-specific equivalent of TextDecoder; it never throws. If you need error detection, use TextDecoder or buffer.isUtf8(bytes) from node:buffer.

Node.js Validating UTF-8

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Decoding in pieces: stream mode

Data from sockets and files arrives in chunks that can end in the middle of a character. With { stream: true } the decoder keeps incomplete bytes for the next call.

Node.js Streaming decode keeps partial characters

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Remember the final decode() with no arguments: it flushes any incomplete trailing sequence. For whole streams the web platform offers TextDecoderStream, which you can pipe a byte stream into, shown in Web Streams in Node.js.

Normalization: one character, several spellings

Unicode allows many characters to be written in more than one way. é is either the single code point U+00E9 or e followed by the combining accent U+0301. They look the same and are canonically equivalent, but as strings they are different, which breaks equality checks, dictionary keys and searches.

Node.js Why two cafés are not equal

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The four forms are NFC (composed, the best default for storage and comparison), NFD (decomposed), and the compatibility forms NFKC and NFKD, which also turn things like ligatures, circled digits and full-width letters into their plain equivalents. Normalise user input once, at the edge, before comparing, indexing or using it as a key. The regular expression with \p{Diacritic} uses Unicode property escapes, which need the u flag.

Case conversion is not trivial either. 'İ'.toLowerCase() and 'ß'.toUpperCase() behave differently from simple ASCII rules, and Turkish has its own dotted and dotless i. For user-facing comparison use localeCompare or Intl.Collator with an explicit locale, and toLocaleLowerCase('tr') when the language matters.

Intl.Segmenter: counting what people see

Intl.Segmenter splits text by grapheme, word or sentence following Unicode rules and a locale. Use it to truncate text safely, to count visible characters and to move a cursor.

Node.js Graphemes, words and sentences

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Segmentation is based on Unicode data, so the exact rules for rare scripts depend on the ICU data bundled with Node. Official Node binaries ship full ICU, so word breaking for languages written without spaces, such as Japanese or Thai, also works; custom builds with --with-intl=small-icu may fall back to English rules. Regular expressions can also help: /\p{L}+/gu finds runs of letters in any script, and /\p{Extended_Pictographic}/u matches emoji.

Base64: bytes as text

Base64 turns any bytes into a string using only 64 safe characters (A-Z a-z 0-9 + /, with = padding), at the price of 33% more size. You meet it in data URLs, email attachments, JSON Web Tokens and HTTP basic authentication. Node offers three ways to use it.

Node.js btoa, atob and Buffer

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

btoa and atob come from browsers and treat a string as raw bytes, one byte per character with code points up to 255; a character like ë still works, but anything above 255 (such as the emoji) throws InvalidCharacterError. That makes them awkward for real text. In Node, prefer Buffer.from(text).toString('base64') and Buffer.from(base64, 'base64'). Both Node and browsers are gaining standard methods directly on Uint8Array, namely toBase64(), fromBase64(), toHex() and fromHex(); check whether your Node version has them before relying on them:

Node.js Do the Uint8Array base64 helpers exist here?

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

On our Node 24.22 server the first four lines print false, false, undefined and undefined: the helpers are not available yet (they are newer than this Node line), so Buffer remains the portable choice. The last line shows the Buffer way of doing hex, which works everywhere.

Reading text from files and other sources

Files carry no label saying which encoding they use. The default for almost everything today should be UTF-8; use readFile(path, 'utf8') and writeFile(path, text) (also UTF-8). For files from older systems that use Windows-1252 or Latin-1, read the bytes and decode with the matching TextDecoder. A byte order mark in UTF-16 files tells you the byte order (FF FE little endian, FE FF big endian).

Node.js Mojibake: UTF-8 read as Latin-1

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Garbled text like ë is called mojibake. It nearly always means "UTF-8 bytes were decoded as a single-byte encoding". The fix is to decode correctly at the source, not to patch the text afterwards. Set Content-Type: text/html; charset=utf-8 on responses and <meta charset="utf-8"> in documents, and use UTF-8 in your database connection, too.

Pitfalls checklist

  • Do not use string.length as a byte count or a visible-character count.
  • Do not slice text that may contain emoji; use code points or Intl.Segmenter.
  • Normalise (NFC) before comparing or using text as a key.
  • Decode with { stream: true } (or a StringDecoder) when chunks arrive separately.
  • Use { fatal: true } or isUtf8 when invalid bytes should be an error.
  • Do not base64 Unicode text with plain btoa; go through UTF-8 bytes or use Buffer.
  • Remember that a BOM can lurk at the start of a file.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What is '🚀'.length in JavaScript?

    The rocket is outside the Basic Multilingual Plane, so UTF-16 stores it as a surrogate pair of two code units. In UTF-8 it takes four bytes.
  2. 2. Which TextDecoder option makes decode() throw on invalid bytes instead of inserting U+FFFD?

    fatal: true turns decoding errors into a TypeError. stream is for chunked input and ignoreBOM keeps the byte order mark.
  3. 3. Why can 'caf\u00e9' === 'cafe\u0301' be false even though both look like café?

    Both are canonically equivalent but not identical strings. Call normalize('NFC') on both before comparing.
  4. 4. What does { stream: true } do in TextDecoder.decode(chunk, { stream: true })?

    With stream: true the decoder remembers a partial multi-byte sequence, so characters split between chunks decode correctly.
  5. 5. Which tool counts the characters a user would see in '👨‍👩‍👧'?

    That emoji is several code points joined by zero-width joiners, but one grapheme. Only Intl.Segmenter gives 1.

Summary

  • Unicode assigns code points; UTF-8 and UTF-16 are encodings. JavaScript strings are UTF-16, files and networks should use UTF-8.
  • Length depends on the unit: code units (length), code points, graphemes (Intl.Segmenter) or bytes (Buffer.byteLength).
  • TextEncoder makes UTF-8 bytes; TextDecoder decodes with options fatal, ignoreBOM and { stream: true }.
  • normalize('NFC') makes equivalent spellings equal; use localeCompare for human sorting.
  • isWellFormed and toWellFormed find and fix lone surrogates.
  • Use Buffer for base64 and hex; btoa and atob handle only one-byte characters.
  • Mojibake means bytes were decoded with the wrong encoding; fix it at the source.

Next: Working with Binary Protocols