Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 39 ยท Getting Started

Text Encoding and Syntax Basics

Understand characters versus bytes, UTF-8, the byte order mark, escaping, line endings and whitespace rules, the basics that make JSON, XML and YAML parse correctly.

Before learning any one format, it helps to understand the layer underneath: how text becomes bytes, and how formats keep their own special characters apart from your data. Most "mysterious" parsing errors come from this layer: a stray byte order mark, a wrong encoding, a missing escape, or Windows line endings.

Characters and bytes

A character is a letter, digit, symbol or emoji. A file stores bytes. An encoding maps characters to bytes. The encoding you should use for every format in this course is UTF-8: it covers every Unicode character, writes plain ASCII as single bytes, and is the default for JSON, TOML and Markdown tooling and for XML and YAML.

Node.js Characters versus bytes in UTF-8

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

One emoji is one character, two JavaScript length units and four bytes. Keep the three numbers apart: size limits are often in bytes, while user-facing limits count characters.

Decoding with the wrong encoding

If a program reads UTF-8 bytes as Latin-1, accented letters turn into garbage ("mojibake").

Node.js Mojibake: the wrong encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The byte order mark

A BOM is the character U+FEFF placed at the start of a file (bytes EF BB BF in UTF-8). Some Windows tools add it. It is invisible, yet JSON parsers reject it, because it is not valid JSON whitespace.

Node.js A BOM breaks JSON.parse

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

When you read files, strip a leading U+FEFF. When you write files, do not add one.

Special characters and escaping

Every format has characters with a reserved meaning. To put one inside your data, you escape it with the format's own mechanism:

Format Reserved How to write it literally
JSON " and \ inside strings \" and \\, also \n, \t, \uXXXX
XML < and & &lt; and &amp; (also &gt;, &quot;, &apos;)
YAML : # and leading -, [, { quote the string
CSV the comma, quotes, newlines wrap the field in "..." and double inner quotes
Markdown *, _, #, backtick backslash, as in \*
Node.js Escaping the same sentence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Never build these strings by pasting user input into a template. Use a serializer, or at least an escape function, every time. That habit is also your first defence against injection problems covered in Parsing, security and tooling.

Whitespace and line endings

  • JSON and XML treat whitespace between tokens as insignificant, so pretty printing is optional.
  • YAML and Markdown give whitespace meaning: indentation shapes the structure, trailing double spaces may mean a line break. YAML forbids tabs for indentation.
  • Line endings differ: Unix uses \n (LF), Windows uses \r\n (CRLF). Most parsers accept both, but diffs, hashes and CSV tools may not. Pick LF in repositories.
Node.js Normalizing line endings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many UTF-8 bytes does the euro sign take?

    The euro sign is U+20AC, which needs three bytes in UTF-8.
  2. 2. Why does JSON.parse fail on text that starts with a BOM?

    U+FEFF is not one of the four whitespace characters JSON allows, so the parser rejects it.
  3. 3. How do you write a literal less-than sign inside XML text?

    XML uses entity references; < stands for the less-than character.
  4. 4. Which of these is whitespace-sensitive?

    YAML uses indentation to express structure.

Summary

  • Use UTF-8 everywhere; characters, UTF-16 units and bytes are three different counts.
  • Strip a leading BOM when reading and never write one.
  • Escape reserved characters with the format's own rules, using a serializer where possible.
  • YAML and Markdown care about whitespace; normalize line endings to LF.

Next: Tools for working with data formats