Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 63 ยท Getting Started with NLP

The NLP Toolkit: NLTK, spaCy, scikit-learn and Transformers

A practical map of the Python NLP ecosystem: what NLTK, spaCy, scikit-learn, Gensim and Hugging Face Transformers do, when to choose each, and how to install them.

Python has a rich ecosystem for language work, and beginners often ask which library to learn first. The honest answer is that they solve different problems and are usually combined. One library teaches concepts, another builds fast production pipelines, a third trains classic models, and a fourth gives you pre-trained neural networks.

This lesson gives you a map. You will see what each major library is for, run the one that works in your browser (scikit-learn), and read short, labelled read-only snippets for the ones that need packages, models or downloads that the in-browser runner does not have. You will also learn why this course builds the core ideas from scratch before it hands the work to a library.

Four libraries, four jobs

NLTK Classic toolkit, corpora and teaching.
spaCy Fast, production-ready language pipelines.
scikit-learn Vectorizers and classic ML models.
Transformers Pre-trained neural models from Hugging Face.

Real projects often mix two or three of them.

1 / 2

NLTK: the classic toolkit

NLTK (the Natural Language Toolkit) was built for teaching and research. It bundles tokenizers, stemmers, a lemmatizer based on the WordNet lexical database, part-of-speech taggers, simple parsers and a large collection of corpora and word lists. Its API exposes each algorithm separately, so you can swap one stemmer for another and compare. That makes it a good companion for learning, and it is still handy for quick experiments, though it is not tuned for speed on large data.

Many NLTK features need extra data files, which you download once by name. Because that requires network access and writing to disk, the snippet below is read-only here.

Python
# pip install nltk
import nltk
nltk.download("punkt_tab")          # tokenizer data, downloaded once

from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer

print(word_tokenize("Dr. Smith can't visit the U.S. today."))
print(PorterStemmer().stem("running"))      # run

spaCy: industrial-strength pipelines

spaCy is designed for building real applications. You load a trained pipeline, pass it text, and get back a Doc object with tokens that carry their lemma, part of speech, dependency relation and, for the whole document, named entities. Pipelines are fast and have a consistent API, and you can add your own components. spaCy chooses good defaults for you, which means less choice than NLTK but quicker results.

A trained pipeline is a separate download of tens of megabytes (the English "small" model is about 12 MB), so it cannot run in this page. The code below shows the typical workflow.

Python
# pip install spacy
# python -m spacy download en_core_web_sm
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is opening an office in Berlin in March.")

for token in doc:
    print(token.text, token.pos_, token.lemma_)

for ent in doc.ents:
    print(ent.text, ent.label_)       # Apple ORG, Berlin GPE, March DATE

You will meet part-of-speech tagging in Part-of-Speech Tagging, entity recognition in Named Entity Recognition and parsing in Syntactic Parsing, each with code you write yourself first and the spaCy equivalent shown read-only.

scikit-learn: vectorizers and classic models

scikit-learn is the general-purpose machine learning library. For NLP it provides vectorizers that turn texts into numeric matrices (CountVectorizer, TfidfVectorizer, HashingVectorizer), classifiers (naive Bayes, logistic regression, linear SVMs), clustering, topic models (NMF, LDA), metrics and Pipeline objects that chain steps together. It runs in your browser, so most of chapters 3, 5 and 10 use it live.

A vectorizer learns a vocabulary with fit and turns documents into a sparse matrix with transform. Words that were not seen during fitting are ignored later, as the last row shows below.

Python CountVectorizer in three lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each row is a document and each column a vocabulary word; the entry is a count. the appears twice in each of the first two documents, which is why the last column contains 2. The result is stored as a sparse matrix because almost all entries are zero in real data.

Here is the complete text classification workflow, a vectorizer plus a classifier chained with make_pipeline. Eight training texts are tiny, so treat the predictions as a demonstration of the API, not of accuracy.

Python A complete text classifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A fitted pipeline accepts raw strings and returns labels, so the vectorizer and the classifier cannot get out of sync. scikit-learn Pipelines for Text covers this in depth.

Gensim and the Transformers library

Two more names come up constantly. Gensim specializes in unsupervised topic modeling and word embeddings such as Word2Vec and fastText, and it is built to stream through corpora larger than memory. The Hugging Face Transformers library gives uniform access to thousands of pre-trained neural models for classification, translation, summarization, question answering and generation. Its simplest interface is the pipeline function.

Python
# pip install transformers torch   (models download from the Hugging Face Hub)
from transformers import pipeline

classifier = pipeline("sentiment-analysis")
print(classifier("I really enjoyed this tutorial."))
# [{'label': 'POSITIVE', 'score': 0.99...}]

These models range from tens of megabytes to many gigabytes, need a GPU for comfortable speed on large models, and download weights from the internet. That is why they appear read-only in this course. Pre-trained Language Models explains how they work and how to use them.

Which library, when

You want to... Reach for
Understand an algorithm step by step Your own code, then NLTK to compare
Tag, parse and extract entities quickly in an application spaCy
Train a baseline classifier on TF-IDF features scikit-learn
Discover topics or train embeddings on a big corpus Gensim (or scikit-learn for small data)
Get state-of-the-art accuracy on a hard task A pre-trained transformer, fine-tuned if needed

A good habit: start with the simplest model that could work, measure it, and move up only when the numbers justify the extra cost. A TF-IDF plus logistic regression baseline trains in seconds and is surprisingly competitive on many tasks.

Why this course builds from scratch first

It is tempting to jump straight to pipeline("sentiment-analysis"). But a library call is a black box, and black boxes fail in ways you cannot diagnose. If you have written a tokenizer, a TF-IDF weighting and a naive Bayes classifier yourself, you understand what breaks when accents are not normalized, why stop-word removal can flip a sentiment, and why a model trained on movie reviews does badly on tweets.

So the pattern in the next chapters is the same: understand the idea with a few lines of plain Python, check it against scikit-learn where that runs in your browser, and see the NLTK, spaCy or Transformers equivalent as a read-only reference. By the time you reach attention and transformers, nothing in the pipeline will be magic.

Installation tip: install libraries inside a virtual environment (see virtual environments) and pin the versions you tested with. NLP libraries download data and models separately from the package, so record those downloads in your project notes too.

More Python NLP libraries

Beyond the big four there is a long tail of libraries. Each fills a niche, and you are likely to meet them in tutorials and job descriptions. None of them runs in the browser editor, so the snippets are read-only.

Library What it is for Good to know
TextBlob Beginner-friendly API on top of NLTK: part-of-speech tags, noun phrases, sentiment (polarity and subjectivity), spelling correction, translation helpers One line per task, but less accurate and slower than modern models
VADER A rule-based sentiment scorer tuned for social media (Valence Aware Dictionary and sEntiment Reasoner) Returns neg, neu, pos and a compound score from -1 to 1; understands capitals, !, emoticons, negation and but
Textacy Higher-level helpers built on spaCy: text cleaning, n-grams, key-term extraction, readability and similarity statistics Works on spaCy Doc objects, so it needs a spaCy model
Stanza Stanford's neural pipeline with models for 60+ languages Modular pipeline: tokenize, multi-word token expansion, POS, lemma, dependency parse, NER and sentiment, each switchable
Flair A PyTorch framework with strong sequence-labeling models Datatypes Sentence and Token; stackable embeddings let you concatenate several embeddings (for example word vectors plus contextual string embeddings) and feed them to a tagger
PyNLPl A toolkit for linguistic data processing Modules such as pynlpl.formats.folia (the FoLiA annotation format), pynlpl.statistics, pynlpl.evaluation, pynlpl.search, pynlpl.textprocessors (with Windower for sliding windows) and a FrequencyList class
Polyglot Multilingual NLP: language detection, tokenization, NER and transliteration for many languages Needs system libraries (ICU) and per-language downloads
KerasNLP (now KerasHub) Keras-based building blocks: tokenizers, Transformer layers and pretrained models Runs on JAX, TensorFlow or PyTorch backends
PyTorch-NLP (torchnlp) Utilities for PyTorch NLP work: dataset loaders, text encoders, samplers, metrics A helper library; you write the model yourself
Python
# pip install textblob vaderSentiment
from textblob import TextBlob
blob = TextBlob("The service was great, but the food was cold.")
print(blob.sentiment)          # Sentiment(polarity=..., subjectivity=...)
print(blob.noun_phrases)       # a list of noun phrases

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
print(SentimentIntensityAnalyzer().polarity_scores("The movie was GREAT!!"))
Python
# pip install stanza
import stanza
stanza.download("en")
nlp = stanza.Pipeline("en", processors="tokenize,pos,lemma,depparse,ner")
doc = nlp("Marie Curie won two Nobel Prizes.")
print([(w.text, w.upos, w.lemma) for w in doc.sentences[0].words])
print(doc.ents)

Polarity and subjectivity. TextBlob's polarity runs from -1 (negative) to 1 (positive). Its subjectivity runs from 0 (factual) to 1 (opinion). Lexicon methods like these are quick baselines and need no training data, which makes them common in beginner sentiment projects. See Sentiment Analysis for the full comparison with trained classifiers.

spaCy beyond tags and entities

spaCy has two more features worth knowing. The text categorizer (textcat) is a pipeline component that you train on labeled documents to assign classes such as topic or sentiment. And spaCy is extensible: you can register your own pipeline components and attach custom attributes to Doc, Span and Token objects, which is how libraries like Textacy add functionality. Speed is a design goal: the core is written in Cython, and nlp.pipe(texts, batch_size=...) processes many texts efficiently, so spaCy fits production services.

How a VADER-style scorer works

You can see the idea behind VADER in a few lines. A lexicon gives each word a valence, then a handful of rules adjust it: boosters like very raise the intensity, a negation within the previous three words flips and softens the value, capitals add emphasis, exclamation marks add a little, and the sum is squashed into the range -1 to 1.

Python A tiny VADER-style sentiment scorer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain "good" sentence scores +0.44; very and two exclamation marks push it to +0.58; shouting GREAT reaches +0.70; not good turns negative (-0.34) instead of positive; and the three negative words give -0.82. The real VADER lexicon has about 7,500 words and more rules (including but, emoticons and slang), but the logic is the same. Because every step is a visible rule, you can explain any score, and that is the main reason to pick a lexicon tool over a neural model when you have no labeled data.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which library is the best fit for building a fast production pipeline that returns tokens, parts of speech and named entities?

    spaCy provides trained pipelines that attach lemmas, tags, parses and entities to each document, with an API aimed at applications.
  2. 2. What does TfidfVectorizer plus LogisticRegression inside make_pipeline give you?

    The pipeline fits the vectorizer and the classifier together and applies both when you predict, so raw text goes in and labels come out.
  3. 3. After fit_transform on three documents, a new document contains the word 'bird', which was not in the training texts. What happens to it in transform?

    A fitted CountVectorizer only has columns for words it saw during fit, so unseen words contribute nothing.
  4. 4. Why does this course write tokenizers and classifiers by hand before using libraries?

    Writing the small versions yourself shows what each step does, which helps you debug and choose well when you use a library.

Summary

  • NLTK is the teaching and research toolkit, spaCy is for fast production pipelines, scikit-learn provides vectorizers and classic models, Gensim handles topic models and embeddings, and Transformers gives access to pre-trained neural models.
  • Only scikit-learn runs live in your browser here; NLTK, spaCy and Transformers need downloads, so their code is shown read-only.
  • A Pipeline of vectorizer plus classifier keeps preprocessing and model consistent.
  • Start with the simplest model that could work and move up only when measurements show you need to.
  • The course builds core ideas from scratch first so that library calls are never black boxes.

Next: Cleaning and Normalizing Text