Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 63 ยท Counting and Classic Representations

Bag of Words

Turn text into numbers with the bag-of-words model. Build a document-term matrix by hand, then use CountVectorizer, n-grams and sparse matrices in Python.

Machine learning models work with numbers, not sentences. Before a classifier can read a review, a search engine can rank a page, or a clustering algorithm can group emails, the text has to become a vector. The simplest way to do that is the bag of words: forget the order of the words, forget the grammar, and just count how many times each word occurs in each document.

The name is literal. Imagine cutting a document into words and dropping them into a bag. You can still say what is in the bag, and how many of each, but not what came first. That sounds like throwing away most of the meaning, yet for topic detection, spam filtering and many other tasks it works surprisingly well, and it is the baseline every fancier method is compared against. You will need your tokenizer and a feeling for word frequencies.

From documents to a matrix

Bag of words has two steps. First, build a vocabulary: the sorted list of every distinct word in your collection (the corpus). Each word gets a fixed column number. Second, represent every document as a row of counts, one count per vocabulary word. Stacking the rows gives the document-term matrix: documents down the side, words across the top.

Two steps

Build a vocabulary Every distinct word gets a column.
Count per document Each document becomes a row of counts.

The rows all have the same length, so models can compare them.

1 / 2

Do it by hand once, so that nothing about it feels like magic.

Python A document-term matrix by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The vocabulary has eight words, so every document is a vector of eight numbers. Document d1 has the twice and cat once. Documents d2 and d3 share the and dog. Because the vectors line up column by column, you can now add them, compare them or feed them to any model. The columns on and sat are zero in d3, which is a first hint of how such matrices look: mostly zeros.

CountVectorizer

Writing this loop yourself teaches the idea, but scikit-learn's CountVectorizer does the same job faster and handles the details: lowercasing, tokenizing with a regular expression, and building the vocabulary. You call fit_transform on the training texts, which learns the vocabulary and returns the matrix in one go.

Python CountVectorizer on three sentences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at what happened:

  • Capital letters and punctuation are gone. The default lowercase=True and the default token pattern, which keeps runs of two or more letters or digits, handled that.
  • The result is a sparse matrix (csr_matrix), not a normal array. .toarray() converts it so we can print it.
  • transform on a new sentence uses the vocabulary learned earlier. The words a and bird are not in it, so they vanish silently. The single-letter a is also dropped by the default token pattern, so the bag of A cat and a bird sat holds only and, cat and sat.

The last point is a rule worth memorizing: fit on training data only, then transform everything else. If you fit on the test set as well, you leak information about it into the model.

Why sparse matrices

A real corpus might have 50,000 documents and 100,000 distinct words. A dense table of that size would need 5 billion numbers; stored as 64-bit floats that is about 40 GB. Yet a typical document uses a few hundred distinct words, so more than 99% of the cells are zero.

A sparse matrix stores only the non-zero entries. The common CSR (compressed sparse row) layout keeps three arrays: the non-zero values, their column indices, and where each row starts. The demo below shows the three arrays for a small matrix, then shows what the common options do to the vocabulary.

Python Sparse storage and vectorizer options

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three arrays describe [0, 2, 0, 0, 1, 0] as the values 2 and 1 in columns 1 and 4, and so on; the empty second row simply starts and ends at the same position. The vocabulary numbers show the price of each option. Adding bigrams roughly doubles the vocabulary on this tiny corpus (9 to 19 words), and on large corpora it grows much faster. Removing English stop words shrinks it to 4, but notice what it removes: not is in scikit-learn's English stop-word list. For a sentiment task that deletes exactly the word that flips a meaning, so think before using stop-word lists.

Useful options

CountVectorizer has a handful of settings you will use constantly:

  • lowercase: on by default. Turn it off when case carries meaning, such as acronyms.
  • ngram_range=(1, 2): include word pairs, so not good becomes a feature. See n-grams.
  • binary=True: record only presence (1) or absence (0), not counts. Often better for short texts, where a word repeated three times is not three times as informative.
  • min_df and max_df: drop words that are too rare or too common. The next lessons cover them in detail.
  • token_pattern or tokenizer: plug in your own tokenization, for example to keep single letters or hashtags.
  • vocabulary: pass a fixed list of words to use as columns.

Strengths and limits

Bag of words is fast, simple, easy to explain and a strong baseline for classification and search. You can even read a linear model's weights and see which words drive a decision.

Its limits are also clear. It ignores word order, so negations and phrases are lost unless you add n-grams. It treats every word as unrelated to every other: car and automobile are different columns, a gap that word embeddings later fill. Raw counts also overweight long documents and very common words, which is exactly what TF-IDF corrects in the next lesson.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a bag-of-words representation keep from a document?

    The bag keeps counts per word and discards order and grammar, so "dog bites man" and "man bites dog" give the same vector.
  2. 2. A new sentence contains a word that was not in the training vocabulary. What does transform do with it?

    The vocabulary is fixed after fit. Unknown words get no column, so they are silently ignored. That is why you fit on training data only.
  3. 3. Why does CountVectorizer return a sparse matrix?

    A document uses only a few hundred of tens of thousands of vocabulary words, so storing just the non-zero values is far smaller than a dense table.
  4. 4. What does ngram_range=(1, 2) add to the features?

    The range means n-grams from size 1 to size 2, so word pairs such as "not good" become features alongside single words.

Summary

  • Bag of words represents each document as a vector of word counts, ignoring order.
  • A vocabulary assigns each distinct word a column; stacking the vectors gives the document-term matrix.
  • CountVectorizer lowercases, tokenizes, builds the vocabulary and returns a sparse matrix; fit on training data, then transform new data.
  • Sparse (CSR) storage keeps only non-zero values, which makes large vocabularies practical.
  • Options such as ngram_range, binary, min_df and stop_words change the features; stop-word removal can delete negations.
  • Raw counts overweight common words, which leads to TF-IDF.

Next, weight words by how informative they are in TF-IDF.