Skip to content
Tutorials Center
My learning

Topics

Lesson 56 of 111 ยท Feature Engineering and Pipelines

Text Features

Turn text into numbers for machine learning: CountVectorizer, TfidfVectorizer, n-grams, sparse matrices, the hashing trick and a small text classifier.

Models need numbers, and text is not numbers. A review, an email or a support ticket has to become a row of numeric columns before any estimator can learn from it. The classic way is the bag of words: ignore the order of words, and describe each document by how often each word occurs in it. It sounds crude, and it is still a strong baseline for spam filters, topic labelling and sentiment.

This lesson builds a bag of words step by step: counting with CountVectorizer, weighting with TfidfVectorizer, adding word pairs (n-grams), understanding why the result is a sparse matrix, using the hashing trick for huge vocabularies, and finishing with a small classifier. The estimators come from the scikit-learn API you already know: vectorizers have fit and transform like any other transformer.

From words to columns

Tokenize Split each text into lowercase words.
Vocabulary One column per word seen in training.
Count Each cell holds how often a word appears.
Weight TF-IDF downweights words that appear everywhere.

The vocabulary is learned from the training texts only.

1 / 2

CountVectorizer: counting words

CountVectorizer does three things in fit: it splits every text into tokens, builds a vocabulary (one entry per distinct token), and remembers it. transform then returns a matrix with one row per document and one column per vocabulary word.

Python A bag of words by hand and with CountVectorizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each row is a document and each column a word, in alphabetical order. The word "the" appears twice in the first two documents, so its column holds a 2 there. In the last line the word "bird" was never seen in training, so it is silently dropped: the vocabulary is frozen after fit. That is exactly what you want, because a model cannot have a weight for a column it never saw.

Useful options of CountVectorizer:

  • lowercase=True (default) folds case.
  • stop_words="english" removes very common words like "the" and "and". It can help, but it can also remove words that matter ("not"), so test it.
  • min_df and max_df drop words that are too rare or too common. min_df=2 removes words that appear in only one document, which shrinks the vocabulary a lot.
  • max_features keeps only the most frequent words.
  • binary=True records presence (0 or 1) instead of counts.

Sparse matrices

A real corpus has tens of thousands of distinct words, but each document uses only a few dozen of them. A dense matrix would be almost all zeros. scikit-learn therefore returns a sparse matrix (SciPy's CSR format), which stores only the non-zero entries and where they are.

Python Why text matrices are sparse

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Most estimators accept sparse input directly (linear models, Naive Bayes, SVMs), but some do not, or they are slow on it. Avoid .toarray() on a big corpus: it can run you out of memory.

TF-IDF: weighting the words

Raw counts give long documents large numbers and treat every word as equally informative. TF-IDF (term frequency times inverse document frequency) multiplies a word's count in a document by a factor that is large for words found in few documents and small for words found everywhere. A word like "the" ends up with a low weight; "refund" in a customer-support corpus ends up with a high one. Then each row is rescaled to length 1 (L2 norm), so long and short documents become comparable.

TfidfVectorizer is a CountVectorizer followed by this weighting, in one step.

Python Counts versus TF-IDF weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With raw counts, "the" is twice as large as "mat" (2 against 1). With TF-IDF the gap shrinks to a ratio of about 1.3 (0.602 against 0.472), because "the" appears in three of the four documents, so its weight is held back, while "mat", which appears nowhere else, is boosted. The word "the" is still the largest entry here only because it occurs twice in this short sentence. On a real corpus, where it appears in almost every document, its weight is much smaller. The last line shows that every row has been rescaled to length 1.

N-grams: a little word order

A bag of words cannot tell "not good" from "good". Adding n-grams, sequences of n consecutive words, brings back some context. With ngram_range=(1, 2) the vocabulary holds single words and word pairs. The price is a much bigger vocabulary, so combine n-grams with min_df.

Python Unigrams and bigrams

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With unigrams only, "good" is the same feature in both documents. With bigrams, "not good" becomes its own feature, which a classifier can learn to treat as negative.

The hashing trick

The vocabulary of CountVectorizer lives in memory and grows with your data. When the vocabulary would be huge, or when the data arrives as a stream, use HashingVectorizer: it runs each token through a hash function and uses the result, modulo a fixed number of columns, as the column index. It stores no vocabulary at all, so it needs no fit, uses constant memory and can process data in chunks (see partial_fit in Linear Models with SGD). The price: you cannot map a column back to a word, and two different words may share a column (a collision).

Python HashingVectorizer: fixed width, no vocabulary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output always has exactly 16 columns, however many words exist. alternate_sign=False keeps the values as plain counts. With a realistic n_features (the default is 2 to the power of 20) collisions are rare enough to ignore.

A small text classifier

Now we put the pieces together. The texts below are generated from small word lists, so the result is deterministic and the task is easy; real texts are much messier. The vectorizer and the classifier go into one pipeline, so the vocabulary is learned from the training texts only.

Python TF-IDF plus logistic regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the coefficients of a linear model on TF-IDF features is a simple way to see what the classifier learned: the largest positive weights are the words that push towards class 1, the most negative towards class 0.

Fit the vectorizer on the training texts only. If you build the vocabulary or the document frequencies from the whole dataset before splitting, information from the test set leaks into the features. A pipeline handles this for you, including inside cross-validation.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does CountVectorizer do with a word that was not in the training texts when you call transform?

    The vocabulary is fixed in fit. Unknown words contribute nothing at transform time, because the model has no weight for a column it never saw.
  2. 2. Why does TfidfVectorizer give a low weight to a word like 'the'?

    IDF is large for words found in few documents and small for words found in many. A word that appears everywhere carries little information about any one document.
  3. 3. Why are bag-of-words matrices stored as sparse matrices?

    Each document uses a tiny part of the vocabulary. A sparse format keeps the non-zero values and their positions only.
  4. 4. What is the main drawback of HashingVectorizer?

    Hashing needs no vocabulary and uses constant memory, but it loses the word-to-column mapping and different words may share a column.
  5. 5. Which setting lets a bag-of-words model tell 'not good' from 'good'?

    Bigrams add word pairs as features, so 'not good' becomes its own column. The other options do not add any order information.

Summary

  • A bag of words describes each text by word counts; CountVectorizer learns the vocabulary in fit and ignores unseen words afterwards.
  • TfidfVectorizer downweights words that appear in most documents and normalizes rows.
  • N-grams add some word order; use min_df and max_features to keep the vocabulary manageable.
  • Text matrices are sparse: avoid .toarray() on large corpora.
  • HashingVectorizer trades interpretability for constant memory and streaming.
  • Put the vectorizer and the classifier in one pipeline so the vocabulary is learned from training data only.

Next: Feature Selection.