Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 63 ยท Cleaning and Tokenization

Stemming and Lemmatization

Reduce words to a common form with a hand-written Porter-style stemmer and a dictionary lemmatizer, then compare speed, accuracy and the mistakes each one makes.

English words change shape: connect, connected, connecting, connection. For a search engine or a classifier counting words, these are four different tokens, even though they clearly share one idea. Stemming and lemmatization are two ways of folding such variants together so the counts add up instead of splitting.

They look similar but work differently. A stemmer chops suffixes with rules and does not care whether the result is a real word. A lemmatizer returns a proper dictionary form (the lemma), and for that it needs a vocabulary and often the part of speech. You will write a small version of each, see where they fail, and learn how to choose. The tokens you feed in come from tokenization.

Two ways to fold word forms

Stem or lemma?

Stemming Rules chop endings. Fast, crude.
Lemmatization A dictionary returns the base word. Slower, cleaner.

studies becomes studi (stem) or study (lemma).

1 / 2

The word stem here means the leftover string after cutting. It may not be a word: happiness becomes happi. That is fine for search and counting, as long as related words land on the same string, but it is awkward when people have to read the output.

A Porter-style stemmer by hand

The best-known stemmer is the Porter stemmer from 1980. It applies several steps of suffix rules, in order, each with a condition such as "the part left over must contain a vowel". The version below is a simplified Porter-style stemmer with the same shape: plural endings first, then -ed and -ing, then derivational endings such as -ization and -ness, and finally some cleanup. It is not the full original, which has more rules and a finer "measure" condition.

Python A simplified Porter-style stemmer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output. connected, connecting and connection all map to connect, studies and studying both give studi, and computation ends up as comput. None of studi, happi or comput is an English word, but each is consistent, so a search for studying can match a document that says studies. The fix-ups in step 1 matter too: hopped becomes hop because the doubled consonant is removed, while hoping becomes hope.

Where stemmers go wrong

Rules cannot know meaning, so a stemmer makes two kinds of mistakes:

  • Overstemming merges words that should stay apart. universe, university and universal all become univers, so a search for one finds the others.
  • Understemming fails to merge words that belong together. Irregular forms are the usual victims: ran stays ran while running becomes run, and better never meets good.

You can tune a stemmer to avoid one error, but only by making the other more common. This trade-off is why real stemmers (Porter, Snowball, Lancaster) differ in how aggressive they are. Lancaster cuts hard; Snowball (the "Porter2" stemmer) is gentler and supports many languages.

Lemmatization with a dictionary

A lemmatizer asks "what is the dictionary form of this word?" That has two parts: a table of irregular forms (mice to mouse, ran to run) and suffix rules whose result is accepted only if it is a known word. The vocabulary check is what keeps the output clean: cities becomes city because city exists, and a rule that would produce citi is rejected.

The lookup also takes the part of speech (POS), because the same string can have different lemmas. We pass the tag in by hand here; real systems predict it, as described in part-of-speech tagging.

Python A dictionary lemmatizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

children, were and mice are solved by the irregular table, which a stemmer cannot do at all. Words that are not in the vocabulary pass through unchanged, which is the safe default: a lemmatizer should not invent words. The price is that someone must maintain a vocabulary of the whole language. Production systems such as WordNet's lemmatizer or spaCy's rule and lookup tables are exactly this idea at scale.

Why part of speech matters

Without a POS tag, a lemmatizer has to guess. Look at three ambiguous words: saw is a noun (a tool) or the past tense of see; leaves is a plural noun or a verb; better is a comparative adjective, but also a verb meaning "to improve".

Python Same word, different lemma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each word maps to different lemmas depending on its tag, and the default tag matters a lot. Many libraries assume every word is a noun unless told otherwise, so running stays running unless you pass VERB. When your lemmas look wrong, check the POS input before anything else.

Which one should you use?

Stemming Lemmatization
Output A stem, often not a word A real dictionary word
Speed Very fast Slower, needs a dictionary
Irregular forms Not handled Handled
Needs POS No Usually helps
Typical use Search and indexing, quick baselines Readable output, linguistics, cleaner features

A reasonable default: use stemming for search and large-scale counting, where speed and recall matter and nobody reads the stems, and lemmatization when the words are shown to people or when irregular forms matter. Both are optional for modern neural models, and subword vocabularies (see subword tokenization) handle word variation another way. As with stop-word removal, test whether it helps your task.

In libraries

NLTK and spaCy provide mature versions. They are read-only here, since the browser runner does not have them.

Python
from nltk.stem import PorterStemmer, SnowballStemmer, WordNetLemmatizer

PorterStemmer().stem("generalization")            # stem
SnowballStemmer("english").stem("running")        # gentler, supports other languages
WordNetLemmatizer().lemmatize("running", pos="v") # needs the POS letter

import spacy
nlp = spacy.load("en_core_web_sm")
[(t.text, t.lemma_) for t in nlp("The children were running")]

spaCy tags each token and picks the lemma from the tag, so you rarely pass the POS yourself.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main difference between a stem and a lemma?

    Stemmers chop endings by rule, so results like "studi" are fine. Lemmatizers return valid base forms such as "study".
  2. 2. Merging "universe" and "university" into one stem is an example of what?

    Overstemming conflates words that have different meanings. Understemming is the opposite error, like leaving "ran" apart from "run".
  3. 3. Why does a dictionary lemmatizer accept a suffix rule's result only if it is in the vocabulary?

    The vocabulary check is what keeps lemmas valid; unknown words are returned unchanged.
  4. 4. A lemmatizer returns "running" unchanged for the word "running". What is the most likely cause?

    Many libraries default to treating words as nouns. Passing the VERB tag gives "run".
  5. 5. Which choice fits a large search index where nobody reads the stored terms?

    For indexing, speed and recall matter and unreadable stems do no harm.

Summary

  • Stemming and lemmatization fold word variants together so counts and matches combine.
  • A stemmer applies ordered suffix rules, is fast and crude, and can overstem or understem.
  • A lemmatizer uses a vocabulary and irregular table to return real base words, usually with a POS tag.
  • Choose stemming for search and speed, lemmatization for readable output; skip both for neural models.

Next, learn how modern models avoid the whole problem in Subword Tokenization: BPE and WordPiece.