Skip to content
Tutorials Center
My learning

Topics

Lesson 128 of 149 ยท Data and the Web

Natural Language Processing with NLTK and spaCy

Learn natural language processing in Python. Tokenize and clean text, stem words, count n-grams, compare documents with TF-IDF, and meet spaCy pipelines and entities.

Computers are good with numbers and awkward with sentences. Yet text is everywhere: reviews, emails, support tickets, news, chat messages. Natural language processing (NLP) is the set of techniques that turn that text into something a program can count, compare, classify and extract facts from. Typical jobs include spam filters, search engines, sentiment analysis, chatbots, keyword extraction and finding the names of people, companies and places in documents.

Two Python libraries are the classic starting points. NLTK (the Natural Language Toolkit) is a teaching-oriented toolbox of tokenizers, stemmers, corpora and algorithms. spaCy is built for production: it runs a full trained pipeline over text and gives you tokens, parts of speech, grammar and named entities in one call. In this lesson you'll learn the standard text-processing steps, run real NLTK code live, build a document-similarity search with TF-IDF, and see how spaCy works.

What runs here and what doesn't. NLTK's rule-based tools (tokenizers, stemmers, frequency counts, n-grams, edit distance) run live in the editors below. Anything that needs a downloaded model or data package, such as part-of-speech tagging, lemmatizing with WordNet or any spaCy model, can't download files in the browser, so those examples are read-only, with typical results described in the text. On your computer, install them with python -m pip install nltk spacy and the downloads shown in each section. Where a live example needs a stand-in for a trained model, it says so.

The pipeline

Most NLP programs follow the same steps. Raw text goes in on the left; numbers or structured facts come out on the right.

flowchart LR
    A["Raw text"] --> B["Tokenize<br/>split into words"]
    B --> C["Normalize<br/>lowercase, stop words, stems"]
    C --> D["Represent<br/>counts, TF-IDF, vectors"]
    D --> E["Use<br/>classify, search, extract"]

Not every project needs every step. Modern models often skip hand-made normalization, but you still need to understand it to read other people's code and to debug your own.

Tokenization

A token is a unit of text: usually a word, a number or a punctuation mark. Splitting on spaces with str.split() is a start, but it leaves punctuation glued to words ("faster!") and doesn't know that "didn't" is really two words. NLTK offers several tokenizers with different rules:

Python Compare tokenizers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the four lines. split() keeps "faster!" and "running,". The Treebank tokenizer, which follows the conventions of a famous research dataset, splits "didn't" into did and n't and separates punctuation. WordPunctTokenizer splits at every punctuation mark, producing didn, ', t. A regular-expression tokenizer lets you define exactly what a word is, here letters and apostrophes, and discards the rest (you saw this style in regular expressions). There is no single right answer; choose by task. For search you may want didn't as one word; for grammar analysis you want did + n't.

NLTK's word_tokenize and sent_tokenize (the popular ones) use the pretrained Punkt sentence model, which needs a one-time nltk.download("punkt_tab") and so can't run here:

Python
import nltk
nltk.download("punkt_tab")                      # one-time download
from nltk.tokenize import sent_tokenize, word_tokenize

text = "Dr. Smith arrived at 9 a.m. He was late."
print(sent_tokenize(text))      # handles abbreviations better than text.split(".")
print(word_tokenize(text))

Cleaning and normalizing

After tokenizing, you usually reduce the variety so that "Cats", "cats" and "cat" count as one thing:

  1. Lowercase everything.
  2. Drop punctuation and numbers if they carry no meaning for the task.
  3. Remove stop words, very common words such as the, a, of and but that add little for tasks like keyword counting. (NLTK has a stop word list in nltk.corpus.stopwords, which needs a download; below we use a short list of our own.) Keep them when meaning depends on them: "not good" without "not" is the opposite.
  4. Stem or lemmatize: reduce words to a base form.

A stemmer chops endings with rules, so it's fast but crude. A lemmatizer looks words up in a dictionary and returns real words ("better" becomes "good" when told it's an adjective). Stemming is available live; lemmatizing needs WordNet data.

Python Stemming and word frequencies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without stemming, cats is counted apart from cat (1 against 4); after stemming they merge into one entry with 5. running becomes run, but happy becomes happi, and mice stays mice because a stemmer cannot know irregular plurals; a lemmatizer would map it to mouse. The two stemmers also disagree sometimes (generously becomes gener with Porter and generous with Snowball). This is the trade-off: stemming is cheap and good enough for search engines, but its output is not always a real word.

Looking at neighbors: n-grams and edit distance

Words gain meaning from their neighbors. An n-gram is a run of n consecutive tokens. Bigrams (n = 2) catch phrases such as "new york" or "not good", and n-gram counts power simple language models and autocomplete. Edit distance counts the single-letter changes needed to turn one word into another, which is the idea behind spell checkers and fuzzy matching:

Python n-grams, autocomplete and edit distance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a toy, but it's the true core of an early autocomplete: count what follows what, then suggest the most frequent continuation. Real systems use far more text, longer n-grams and neural models.

Tags and entities (a stand-in)

Two classic tasks need a trained model: part-of-speech (POS) tagging (is this word a noun, verb, adjective?) and named entity recognition (which spans are people, organizations, places, dates?). Real taggers learn from many thousands of labelled sentences, so they cope with ambiguity: "book" is a noun in "read a book" and a verb in "book a flight".

With NLTK, after the downloads, tagging is two lines (read-only):

Python
import nltk
nltk.download("punkt_tab")
nltk.download("averaged_perceptron_tagger_eng")

tokens = nltk.word_tokenize("I will book a flight and read a good book.")
print(nltk.pos_tag(tokens))
# Typical output: [('I', 'PRP'), ('will', 'MD'), ('book', 'VB'), ('a', 'DT'), ('flight', 'NN'), ...]

The tags come from the Penn Treebank set (NN noun, VB verb, JJ adjective, DT determiner, and so on). To show the idea without a model, here's a deliberately naive stand-in: a word list plus two rules. It works on the sentences we give it and fails on plenty of others, which is exactly why real taggers are statistical:

Python A toy tagger and entity finder (not a real model)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even the first line shows a flaw: Maria is tagged NOUN, not PROPN, because the capital-letter rule skips the first word of a sentence (a capital there proves nothing). The failures at the end are the lesson: spelling clues and word lists can't resolve ambiguity, because the tag depends on the surrounding words. Statistical models learn those patterns from data.

Comparing documents: bag of words and TF-IDF

To feed text to a machine learning model or a search engine, you convert each document to a vector of numbers. The simplest representation is a bag of words: one column per vocabulary word, each cell the count of that word. Counts have a flaw: common words dominate. TF-IDF (term frequency times inverse document frequency) down-weights words that appear in many documents and boosts words that are distinctive for one document. Two documents can then be compared by the angle between their vectors (cosine similarity), where 1 means the same direction and 0 means no shared words.

Python A tiny search engine with TF-IDF

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each query finds the right documents because they share distinctive words with it. The method has an obvious limit: it matches words, not meaning. A query for "feline" would find nothing. Representations that capture meaning (word vectors and transformer models) address that, and are covered in deep learning. For a complete worked example that classifies text with these ideas, see the sentiment analysis project and machine learning basics.

spaCy: a full pipeline in one call

spaCy loads a trained pipeline, a package that bundles a tokenizer, a tagger, a dependency parser, a lemmatizer and an entity recognizer. You download a model once, load it, and call it on text to get a Doc object:

Terminal
python -m pip install spacy
python -m spacy download en_core_web_sm        # a small English model
Python
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Maria joined Google in Berlin in March 2021. She loves the city.")

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.dep_, token.is_stop)

for entity in doc.ents:
    print(entity.text, entity.label_)
# Typically: Maria PERSON, Google ORG, Berlin GPE, March 2021 DATE

for sentence in doc.sents:
    print(sentence.text)

Each Token carries many attributes: text, lemma_ (the base form), pos_ (a coarse part of speech such as NOUN or VERB), dep_ (its grammatical role, such as subject or object), and flags such as is_stop and is_punct. Attributes ending in an underscore give readable strings; the same name without the underscore gives an integer ID. doc.ents lists the named entities with labels like PERSON, ORG, GPE (countries, cities), DATE and MONEY, and doc.sents splits sentences using the parse, which is more reliable than splitting on periods.

Two features make spaCy practical at scale. nlp.pipe(texts, batch_size=50) processes many documents efficiently, much faster than calling nlp(text) in a loop. And the rule-based Matcher and PhraseMatcher find patterns of tokens (for example "an adjective followed by the word service") without any training. The exact tags and entities depend on the model version, so test your pipeline on your own data before you rely on it.

NLTK or spaCy?

NLTK spaCy
Best for learning, research, linguistic experiments production pipelines
Style a toolbox of separate functions one pipeline, objects with attributes
Speed slower fast (compiled core)
Models classic, data via nltk.download trained pipelines via spacy download
Strong at tokenizers, stemmers, corpora, algorithms entities, dependencies, large-scale text

They work well together: many projects use NLTK for a stemmer or a quick experiment and spaCy for the pipeline. For the hardest tasks (summarizing, translating, answering questions), pretrained transformer models, usually used through the Hugging Face libraries, outperform both.

Common mistakes

  • Cleaning too aggressively. Removing stop words or punctuation can destroy meaning in sentiment tasks ("not", "!!!").
  • Skipping the downloads. LookupError: Resource punkt_tab not found means an NLTK data package is missing; the error message prints the exact nltk.download(...) call.
  • Treating stems as words. "studi" is fine inside a counting pipeline, not for display.
  • Calling nlp(text) in a loop for thousands of texts. Use nlp.pipe.
  • Trusting a model blindly. Taggers and entity recognizers make mistakes, especially on text unlike their training data (slang, product names, other languages).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a stemmer do?

    Stemmers such as Porter's apply suffix-stripping rules. That is fast but can produce non-words like studi. Lemmatizers use a dictionary and return real words.
  2. 2. Why might you keep stop words when analyzing sentiment?

    Stop-word removal helps for keyword counting, but small words like not can carry the sentiment.
  3. 3. In TF-IDF, which words get the highest weight in a document?

    The inverse document frequency part reduces the weight of words found everywhere and rewards words that distinguish one document from the rest.
  4. 4. What is a bigram?

    An n-gram is a run of n consecutive tokens. Bigrams capture short phrases such as "green tea" and are the basis of simple predictors.
  5. 5. In spaCy, how do you process thousands of texts efficiently?

    nlp.pipe streams documents through the pipeline in batches, which is much faster than separate calls.

Summary

  • NLP turns text into data: tokenize, normalize, represent, then classify, search or extract.
  • NLTK's tokenizers differ in how they treat punctuation and contractions; choose by task.
  • Lowercasing, stop-word removal and stemming reduce variety but can lose meaning; lemmatizers return real words.
  • n-grams capture word order, edit distance powers spell checking, and TF-IDF with cosine similarity gives a simple document search.
  • POS tagging and named entity recognition need trained models (NLTK downloads, spaCy pipelines); spaCy returns tokens, lemmas, parts of speech, dependencies and entities in one call.
  • Use NLTK to learn and experiment, spaCy for production, and transformer models for the hardest language tasks.

Next: Image Processing with Pillow, where the data you work with is pictures instead of words.