Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 63 · Getting Started with NLP

Levels of Language: Characters to Meaning

See language the way an NLP system does: characters and encodings, morphology, syntax, semantics and pragmatics, with Unicode normalization examples you can run.

Linguists describe language in layers. At the bottom are sounds or characters, then words and their parts, then sentences and their structure, then meaning, and at the top the intention behind what someone says in a particular situation. NLP systems are organized the same way. A tokenizer works near the bottom, a parser in the middle, a question-answering system near the top.

Knowing the layers helps you diagnose problems. If a search engine cannot find "café" when you type "cafe", that is a character-level issue. If a model confuses "dog bites man" with "man bites dog", that is a structure-level issue. If a chatbot answers "Can you pass the salt?" with "Yes", that is a pragmatics issue. Different layers need different tools.

The layers of language

Characters Letters, digits, symbols and their encodings.
Morphology Words and their parts, like walk + ing.
Syntax How words combine into sentences.
Meaning Semantics and pragmatics.

Errors at a low layer spread upward.

1 / 2

Characters, code points and encodings

Computers store numbers, not letters. Unicode is the agreement that gives every character a number called a code point: A is 65 (written U+0041), é is 233, 你 is 20320 and the grinning-face emoji is 128512. Python's str is a sequence of code points, and the unicodedata module in the standard library can tell you the official name and category of each one.

An encoding says how code points are turned into bytes for storage or transfer. UTF-8 is the dominant one on the web: ASCII characters use one byte, accented Latin letters two, most Chinese characters three, and emoji four. Older encodings such as Latin-1 use one byte per character and cannot represent most of the world's scripts.

Python Code points, names and categories

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The category codes are useful in cleaning code: Ll is a lowercase letter, Lo is another letter (such as a Chinese character), So is a symbol and Mn is a nonspacing mark that attaches to the character before it.

Bytes and mojibake

When bytes are decoded with the wrong encoding you get garbled text known as mojibake. It is one of the most common real-world data problems, and you can reproduce it in two lines.

Python Length in characters versus bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The text café is four characters but five UTF-8 bytes, and decoding those bytes as Latin-1 turns the single é into the two-character junk é. If you ever see é or ’ in a dataset, someone decoded UTF-8 as a one-byte encoding. The fix is at the source: always read and write files with encoding="utf-8" and be explicit about it.

Normalization: one look, several spellings

The same visible text can be built from different code points. The letter é can be one code point (U+00E9) or an e followed by a combining acute accent (U+0301). They look identical on screen but are different strings, so a search or dictionary lookup comparing them fails.

Unicode defines four normalization forms:

Form What it does
NFC Compose: combine base letter and accent into one code point where possible
NFD Decompose: split into base letter plus combining marks
NFKC Compose and also replace "compatibility" characters (ligatures, circled digits, full-width letters) with plain equivalents
NFKD Decompose plus compatibility replacement
Python Two spellings of the same word

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two spellings print the same but compare as different, with lengths 4 and 5. After any of the four forms they compare equal. NFKC also turns the "fi" ligature, the circled digit, the Roman numeral and the full-width letter into plain fi, 1, IV and f. A safe default for text going into a model is NFKC, though it is lossy: ① and 1 become the same thing, which may or may not be what you want.

Removing accents on purpose

Search engines often want "creme brulee" to match "crème brûlée". The standard trick is to decompose with NFD and drop the combining marks (category Mn).

Python Strip accents with NFD

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treat this as a tool, not a default. In Spanish, año (year) and ano (anus) differ only by the tilde; in German, dropping the umlaut can change a word's meaning. Cleaning and Normalizing Text discusses when to use it.

Morphology: the parts of words

Morphology studies how words are built from smaller meaningful parts called morphemes. Walked is walk plus the past-tense marker -ed. Unhappiness is un- plus happy plus -ness. English adds a modest amount of structure to its words, but many languages add far more: Turkish and Finnish can pack an entire English sentence into a single word.

NLP cares about morphology because it creates many surface forms of the same idea. Walk, walks, walked, walking are four different strings, yet in most tasks they should count as related. Two classic responses are stemming and lemmatization (Stemming and Lemmatization); a more modern response is breaking words into subword pieces (Subword Tokenization).

Python A naive suffix splitter

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

It splits the regular words correctly and leaves sing and bed alone thanks to the length guard. Without that guard, sing would be cut into s + ing, which shows how quickly rules need exceptions.

Syntax: how words combine

Syntax is about structure. The words dog, bites and man in different orders produce different meanings: "Dog bites man" and "Man bites dog" contain exactly the same words. A model that ignores order, like the bag-of-words methods in chapter 3, cannot tell them apart. Syntax also explains why "colorless green ideas sleep furiously" feels grammatical even though it means nothing.

Two classic ways to describe structure are constituency (nested phrases such as noun phrase and verb phrase) and dependency (arrows from each word to the word it modifies). Both come back in Syntactic Parsing.

Semantics and pragmatics

Semantics is the literal meaning of words and sentences. It includes word senses (the two meanings of bank), relations between words (dog is a kind of animal) and who did what to whom. Chapter 7 shows how word embeddings capture some of this by representing a word through the company it keeps.

Pragmatics is meaning in context: what the speaker intends. "Can you pass the salt?" is grammatically a yes/no question but is really a request. "It's cold in here" may be a request to close the window. Sarcasm, politeness and implied meaning live here, which is why they are the hardest part for current systems.

Why the layers matter for you

Mistakes cascade. If the character layer is wrong (two encodings of the same word), your counts are split. If tokenization is wrong, every later step sees the wrong units. This is why the next chapters spend real time on cleaning and tokenization before any clever modeling.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. A search for 'cafe' fails to find a page containing 'café' stored as 'cafe' plus a combining accent. At which level is the problem?

    The two strings use different code points for the same visible text. Normalizing the characters (and optionally stripping accents) solves it.
  2. 2. What does unicodedata.normalize('NFKC', text) do that NFC does not?

    The K stands for compatibility. NFKC composes like NFC and additionally maps characters such as the fi ligature or circled digits to their plain forms.
  3. 3. Decoding the UTF-8 bytes of 'café' as Latin-1 produces 'café'. What is this called?

    Mojibake is text garbled by decoding bytes with the wrong encoding. It is fixed by decoding with the right codec at the source.
  4. 4. Why can a bag-of-words model not tell 'dog bites man' from 'man bites dog'?

    Bag of words keeps counts but discards order, so the two sentences look identical, although their syntax gives them opposite meanings.

Summary

  • Language works in layers: characters, morphology, syntax, semantics and pragmatics; NLP tools target different layers.
  • Unicode gives each character a code point; an encoding such as UTF-8 turns code points into bytes, and decoding with the wrong codec produces mojibake.
  • The same visible text can have several code-point spellings; unicodedata.normalize (NFC, NFD, NFKC, NFKD) makes them comparable.
  • Accent stripping (NFD plus dropping Mn characters) helps matching but can change meaning in some languages.
  • Morphology creates many forms of one word, syntax gives word order its meaning, and pragmatics is the hardest layer for machines.

Next: Working with Text in Python for NLP