Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 73 ยท Projects

Project: Sentiment Classifier

Build a sentiment classifier in NumPy: bag-of-words and embedding networks versus logistic regression, then use error analysis to find what they miss.

Can a program tell whether a movie review is positive or negative? This is the "hello world" of text classification, and it is a perfect project for seeing how deep learning handles language: words have to become numbers before a network can touch them. In this project you build a small review corpus, turn it into numbers in two different ways, train two networks in NumPy, and compare them with a logistic-regression baseline.

Then comes the part that separates a project from a demo: error analysis. Instead of staring at one accuracy number, you will look at the reviews each model gets wrong and discover a pattern, the word "not". Fixing the problem teaches more than any amount of tuning would.

Two ways to turn text into numbers

Bag of words One 0 or 1 per vocabulary word; order is lost.
Embeddings Each word gets a learned vector; average them.
Bigrams Count word pairs so "not great" is its own feature.
1 / 2

A corpus with a catch

Real review datasets are large and need downloading, so we generate one. Each review has three clauses such as the plot was not excellent or i found the cast tedious. A clause is positive or negative depending on its adjective, and a "not" in front of the adjective flips it: "not excellent" is bad, "not boring" is good. About 30 percent of the clauses are neutral filler like the theater was half full. The label is the overall sign of the review, and we throw away the rare ties.

The vocabulary is small and the grammar is regular, so the problem would be trivial if word order did not matter. But it does, and that is the catch we want the models to expose. Roughly 45 percent of the reviews contain at least one negation.

We start with the baseline every text project should have: logistic regression on binary word-presence features, built by scikit-learn's CountVectorizer. We also try the same model with bigrams, which are pairs of neighbouring words, so that not excellent becomes a feature of its own.

Python The corpus and a logistic-regression baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table closely. With single words, logistic regression scores only about 70 percent overall, and the split explains why: on reviews without a negation it is almost perfect (99 percent), and on reviews with one it is worse than a coin flip, 40 percent. The model sees excellent and not as unrelated words, so it cannot know that the first was flipped by the second. Adding bigrams lifts the same model to nearly 98 percent. Hold on to that number: it is the bar the neural networks must clear.

Two neural networks

Neural networks need numbers, so first we tokenize (split into words) and build a vocabulary that maps each word seen in training to an index. Then there are two classic ways to feed the words in:

  1. Bag of words (BoW). A vector with one slot per vocabulary word, 1 if the word occurs in the review. The network is an MLP: vocabulary -> 16 hidden ReLU units -> 1 sigmoid output. It has no idea of word order, but the hidden layer can learn interactions such as "not and excellent both present".
  2. Embeddings. Each word has a learned vector of 8 numbers (see Embeddings). A review is represented by the average of its word vectors, which then goes through a hidden layer to the output. The vectors are trained by backpropagation along with everything else, so the network decides for itself which words should be near each other. The embedding table is the first layer of nearly every modern language model.

Both are trained with binary cross-entropy and Adam. The backward pass for the embedding table is the interesting bit: only the rows of words that appeared in the batch get a gradient, so we scatter-add the gradient with np.add.at.

After training, the example reports accuracy overall and split by whether the review contained a negation, and projects the 8-dimensional word vectors down to 2 dimensions with an SVD so that you can look at them.

Python Bag-of-words network versus embedding network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both networks reach almost perfect accuracy on reviews without a negation, so they have no trouble with the vocabulary itself. Their problem is the reviews with a "not": the bag-of-words MLP is right only about half of the time, and the embedding network is clearly better at about 68 percent, even though it also averages away the order. The reason is that the hidden layer can learn "not is present, so lean the other way", which helps in many reviews (not all: a review may contain one negated and one plain clause). The word-vector plot shows the embedding doing its job: green positive adjectives and red negative adjectives separate along one direction, and the neutral nouns sit in the middle.

Reading the mistakes

At 84 percent the embedding network might look acceptable, and many projects stop here. A better habit is to read the errors. Below we list a few test reviews the network got wrong, count how many contained a negation, and then test the hypothesis the errors suggest: word order matters, so give the model features that capture neighbours.

The fix is to switch the bag-of-words input from single words to single words plus bigrams. The vocabulary grows from about 50 to about 450 features, but each review still gets a plain vector, so the very same MLP code works unchanged.

Python Error analysis and the bigram fix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All 79 of the embedding network's mistakes are reviews with a negation, and when you read them you can find the flipped clause that the averaging blurred out (often next to a clause with the opposite sign). The bigram features fix the failure: the same kind of MLP now reaches 99.4 percent, a little above logistic regression with bigrams at 97.8 percent. This is a typical result. The neural network was not the fix. A better representation was.

On a real corpus the network would be much larger, and a recurrent network or a transformer would read the words in order and learn about "not" by itself. See Recurrent Neural Networks and The Transformer Block. The workflow stays the same: baseline first, then models, then reading the errors.

Lessons from the project

  • A strong linear baseline is hard to beat. On this data, logistic regression with bigrams comes within two points of the best network and trains in a blink.
  • Representation matters more than architecture. Bag-of-words discards order; averaging embeddings discards it too. Capturing neighbours was worth more than any hyperparameter.
  • Split your metric. Overall accuracy hid the fact that one subgroup (reviews with "not") was failing. The same habit finds bias in real systems, as discussed in Responsible Deep Learning.
  • Check how the test set was made. Our test reviews come from the same generator as the training set. Real reviews bring sarcasm, misspellings and words never seen in training. Unknown words get no embedding at all in this code: a real pipeline needs an "unknown" token or subword tokenization.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does a single-word bag-of-words model fail on the review 'the plot was not excellent'?

    Bag-of-words records which words occur but not which words are next to each other, so the negation is invisible to a linear model.
  2. 2. How is an embedding network's review representation computed in this project?

    Each word index looks up a trainable vector, and the review is the mean of those vectors, which a small MLP turns into a probability.
  3. 3. Why is np.add.at used in the embedding backward pass?

    Fancy-index assignment would keep only one of the repeated contributions. np.add.at accumulates all of them.
  4. 4. What did error analysis reveal in this project?

    Reading the mistakes and grouping them turned a vague 84 percent into a concrete hypothesis that bigram features then confirmed.
  5. 5. Which conclusion is best supported by the final comparison?

    With bigram features both the linear model and the MLP reach 98 to 99 percent, while the embedding network without them stays at 84 percent, so the representation was the main lever.

Summary

  • Text must be tokenized and mapped to numbers: a vocabulary of word indices feeds either bag-of-words vectors or an embedding table.
  • Always start with a baseline such as logistic regression on word-presence features.
  • An embedding network averages learned word vectors; its gradient touches only the rows of the words that occurred (np.add.at).
  • Split the metric by subgroup: overall accuracy hid a failure on reviews with negations.
  • Error analysis turns a number into a hypothesis; bigram features confirmed this one and lifted both models to 98 to 99 percent.
  • Representation often matters more than model size.

Next project: Time-Series Forecaster.