Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 63 ยท Counting and Classic Representations

Feature Hashing and Large Vocabularies

Handle huge vocabularies in NLP. Learn how HashingVectorizer maps words to buckets, how collisions work, and how min_df, max_df and max_features shrink features.

A bag of words needs a vocabulary: a dictionary from every word to its column number. On a small data set that is fine. On a large one it becomes a problem. Web text contains millions of distinct strings once you count names, typos, numbers, URLs and word pairs. The vocabulary must be kept in memory, built in a first pass over the data, saved with the model, and shared between machines. It is also useless for words that appear after training.

There are two standard ways out. You can prune the vocabulary by dropping words that are too rare or too common. Or you can hash words straight to column numbers so that no vocabulary is needed at all. This lesson covers both, and how to pick between them. It builds on Bag of Words and TF-IDF.

The hashing trick

A hash function turns any string into a number, and always gives the same number for the same string. If you divide that number by the number of columns you want, the remainder is a column index. This is the entire idea: instead of looking up "apple" in a dictionary, compute hash("apple") % n_features and use the result as its column.

Dictionary versus hash

Vocabulary Look the word up in a stored dictionary.
Hashing Compute the column from the word itself.

Hashing needs no dictionary and no first pass over the data.

1 / 2

The demo uses zlib.crc32 as the hash. Python's built-in hash() is deliberately randomized for strings from one run to the next, so it must never be used for this; a stable hash is essential if the model is saved and loaded later. Watch what happens as the number of buckets changes.

Python Hashing words into buckets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 4 buckets, seven words must share: apple and cherry both land in bucket 0. With 16 buckets, elderberry and fig collide in bucket 5. With 1,024 buckets all seven words get separate buckets. That is a collision: two different words share one column and their counts are added together. The model cannot tell them apart.

HashingVectorizer

Scikit-learn's HashingVectorizer applies the trick to the whole bag-of-words pipeline. It lowercases and tokenizes like CountVectorizer, but sends each token to a bucket using the MurmurHash function. Its main settings:

  • n_features: the number of columns (the default is 2 to the power 20, about a million).
  • alternate_sign: by default it gives each word a positive or negative sign (also derived from the hash), so that collisions tend to cancel out instead of piling up.
  • norm: L2 normalization by default; set None to see raw counts.
Python A hashed bag of words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first sentence, the cat sat on the mat, has six tokens and five distinct words, but only four buckets are non-zero (the counts are 1, 2, 1 and 2). Two words collided. In the signed version, the same two words had opposite signs, and their counts cancelled to 0 in one bucket. That is the point of alternate_sign: collisions add noise that averages out instead of a bias that accumulates, which helps linear models.

Two more details from the output. A fresh HashingVectorizer gives exactly the same matrix, because there is no state to learn: you do not call fit, you just call transform. And there is nothing like get_feature_names_out; the vectorizer never stored a word.

That statelessness is the main practical benefit. You can stream a data set too big for memory in chunks, vectorize each chunk independently, train with partial_fit, and later vectorize new documents on a different machine with the same settings.

How many collisions?

If m distinct words are thrown into n buckets at random, the expected number of occupied buckets is n * (1 - (1 - 1/n)^m). The example generates 1,686 made-up words and measures the real number against that formula.

Python Measuring collisions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 256 buckets, every bucket is used and 1,430 of the 1,686 words share one. At 16,384 buckets, 97 words still share. Collisions never fully disappear, but they matter less than you might fear: the words that share a bucket are usually rare, and a classifier can still rely on the many other features. In practice, set n_features to a power of two much larger than your expected vocabulary (hundreds of thousands to millions for real text), and test whether doubling it changes your accuracy.

The costs of hashing are real, and they are the reason it is not the default choice:

  • You cannot map a column back to a word, so you cannot read off the most influential words from a model.
  • TfidfVectorizer has no hashing mode. You can chain HashingVectorizer with TfidfTransformer to get IDF weights, at the price of keeping the document frequencies.
  • Collisions add a small amount of noise.

Pruning the vocabulary

If you can afford a vocabulary, shrinking it often improves both speed and accuracy. Three CountVectorizer and TfidfVectorizer settings do this:

  • min_df: ignore words that appear in fewer than this many documents. An integer is a document count; a float is a fraction. Words seen once are typically typos or names, and a model cannot learn from them.
  • max_df: ignore words that appear in more than this fraction of documents. This removes corpus-specific stop words without a list.
  • max_features: keep only the most frequent words by total count.
Python min_df, max_df and max_features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Twelve words shrink to five with min_df=2, which drops everything that appears in one document, including the rare zyzzyva. max_df=0.7 removes only the, which is in four of five documents (80%). Combined, the two leave four words. max_features=3 keeps the three most frequent words in the whole corpus, here the, cat and on (words with the same total count are ordered alphabetically).

Keep in mind that pruning is done on the training set during fit. Never choose thresholds by looking at the test set.

Which should you use?

Situation Choice
Small or medium data, you want interpretable weights CountVectorizer or TfidfVectorizer with min_df
Data too large for memory or streaming input HashingVectorizer
Many unseen words after deployment (names, URLs, hashtags) HashingVectorizer, or character n-grams
You need to explain predictions word by word A real vocabulary

For the rest of this course we mostly use real vocabularies, because they keep the weights readable.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does HashingVectorizer do instead of storing a vocabulary?

    The column index is the word's hash modulo n_features. Because that depends only on the word, no dictionary has to be built or stored.
  2. 2. Why is Python's built-in hash() unsuitable for feature hashing?

    Python randomizes string hashing per process. A stable function such as MurmurHash or CRC32 gives the same column every time.
  3. 3. What is a hash collision in this setting?

    With fewer columns than distinct words, some words must share a column. The model then cannot distinguish them.
  4. 4. Which setting removes words that appear in fewer than 5 documents?

    min_df is the minimum document frequency. An integer means a count of documents, so min_df=5 drops anything seen in fewer than five.
  5. 5. What is a real disadvantage of feature hashing?

    The hash is one-way and collisions merge words, so columns cannot be mapped back to words, which hurts interpretability.

Summary

  • A vocabulary-based bag of words needs memory, a first pass and cannot represent unseen words.
  • The hashing trick sends each word to column hash(word) % n_features; it is stateless, memory-bounded and handles new words.
  • Collisions are unavoidable; choose a large power of two for n_features, and use alternate_sign so collisions tend to cancel.
  • Use a stable hash, never Python's randomized hash().
  • min_df, max_df and max_features prune a real vocabulary; set them on training data only.
  • Choose hashing for huge or streaming data; choose a real vocabulary when you need interpretability.

Next, compare documents with Document Similarity and Cosine Distance.