Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 40 ยท Pipelines and Feature Engineering

Text Feature Extraction and Text Classification

Turn text into numbers with scikit-learn CountVectorizer, TfidfVectorizer and HashingVectorizer, then build a text classification pipeline with Naive Bayes and logistic regression.

Models need numbers, but much of the world's data is text: reviews, tickets, headlines, emails. The standard approach is the bag of words: ignore word order, count how often each word occurs, and treat each document as a long row of counts. It is crude and surprisingly effective for classification, topic search and spam filtering.

scikit-learn's sklearn.feature_extraction.text module builds those matrices for you, as sparse arrays (most entries are zero, and only non-zeros are stored).

CountVectorizer

It splits text into tokens, builds a vocabulary from the training documents, and counts.

Python Bag of words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each column is a word and each row a document. Words absent from the training vocabulary are ignored when transforming new text. Useful options:

  • lowercase=True (default), stop_words="english" to drop very common words.
  • ngram_range=(1, 2) to add word pairs ("not good") that capture a little order.
  • min_df=2 drops words in fewer than two documents; max_df=0.9 drops words in over 90%.
  • max_features=5000 keeps only the most frequent terms.
  • binary=True records presence instead of counts.

Text to numbers

Tokenize Split text into words.
Vocabulary One column per word.
Count or weight Counts or TF-IDF values.
Sparse matrix Mostly zeros, stored compactly.

Fit the vectorizer on training text only.

1 / 2

TF-IDF: weighting by rarity

Raw counts overrate words like "the". TF-IDF multiplies a term's frequency in the document (TF) by the log of how rare it is across documents (IDF), then normalizes each row. Words that appear everywhere get low weights; distinctive words get high weights.

Python TfidfVectorizer: top words per document

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rows have unit length, so dot products between rows are cosine similarities, the basis of simple document search.

Python Tiny search engine with cosine similarity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A text classification pipeline

Put the vectorizer and a classifier in one pipeline. Linear models (LogisticRegression, LinearSVC, SGDClassifier) and MultinomialNB (Naive Bayes) are strong, fast choices for sparse text.

Python Sentiment classifier on a tiny corpus

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reviews are generated from word lists so the example is self-contained; real projects use thousands of genuine documents. A nice property of linear models on bag-of-words: the coefficients are interpretable.

Python Which words drive the prediction?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Large vocabularies: HashingVectorizer

CountVectorizer stores a vocabulary dictionary, which can be huge. HashingVectorizer applies a hash function to words to pick a column, so it needs no vocabulary, uses constant memory and can process streams in batches (with partial_fit models such as SGDClassifier). The trade-offs: no way to map columns back to words, and rare hash collisions.

Python HashingVectorizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical tips

  • Fit vectorizers only on training text; inside a pipeline this is automatic.
  • Start simple: TF-IDF + a linear model is a hard baseline to beat.
  • Clean text lightly (lowercasing is built in). Heavy stemming is rarely needed.
  • Combine text and numeric columns with a ColumnTransformer: ("txt", TfidfVectorizer(), "review"), passing a single column name string (not a list) so the vectorizer gets a 1-D input (see pipelines).
  • For meaning beyond word counts, use pretrained embeddings or transformer models outside scikit-learn, then feed their vectors in as features.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is each row of a CountVectorizer matrix?

    Rows are documents and columns are vocabulary terms.
  2. 2. What does TF-IDF do to a word that appears in nearly every document?

    The inverse document frequency shrinks weights of words that carry little discriminating information.
  3. 3. Why fit the vectorizer on training documents only?

    Learning the vocabulary or IDF from test text would leak information, just like fitting a scaler on everything.
  4. 4. What is the main advantage of HashingVectorizer?

    It hashes words to column indices, so no vocabulary dictionary is needed.

Summary

  • Bag-of-words turns documents into sparse count matrices; TF-IDF reweights by rarity.
  • ngram_range, min_df, max_df and stop_words control the vocabulary.
  • TF-IDF plus Naive Bayes or a linear model is a fast, strong baseline, and its coefficients show which words matter.
  • HashingVectorizer handles huge or streaming data without a vocabulary.

Next: Saving Models with joblib