Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 45 ยท Projects

Project: Sentiment Classifier

Build a sentiment classifier: tokenize, build a vocabulary, pad, embed, pool and train. PyTorch code is read-only, with live NumPy and scikit-learn versions.

Sentiment classification, deciding whether a review is positive or negative, is the standard first project in natural language processing. It teaches the pipeline that every text model shares: split text into tokens, map tokens to integers with a vocabulary, pad the batch into a rectangle, turn integers into vectors with an embedding table, combine the vectors into one summary per text, and put a classifier on top. The model at the end is tiny; the pipeline around it is what you will reuse for every larger one.

PyTorch cannot run on this site, so the PyTorch model and training loop are read-only. The live editors work on a synthetic dataset of 2,000 reviews that a short generator writes (so there is no download, and we know the true label of every review). The model that trains live is a NumPy re-implementation of the same architecture with hand-written gradients, and a scikit-learn bag-of-words model shows what a better text representation buys. Read Embeddings and Text Data first if nn.Embedding is new to you.

From text to a prediction

Tokenize Split text into words.
Vocabulary Give every word an integer id.
Pad Make a rectangular batch plus a mask.
Embed One trainable vector per id.

Then pool the vectors, add a linear layer, apply a sigmoid.

1 / 2

The data

The generator builds each review from two or three clauses such as "the plot was really poor" or "honestly the cast was not very great". Each clause names a positive or negative adjective, and in about one clause out of five the adjective is negated with "not", which flips its meaning. The label is the sign of the total: reviews whose clauses cancel out exactly are discarded, so the label is always well defined. That makes the dataset a good teaching tool. A model that only counts good and bad words will get most reviews right, but it will fall short of 100% because it cannot see negation.

Step 1: tokenize, build a vocabulary, pad

Tokenization here is text.lower().split(). Real projects use a library tokenizer (word pieces or byte-pair encoding), because splitting on spaces cannot handle punctuation, rare words or other languages, but the contract is the same: text in, list of tokens out. The vocabulary maps each token to an integer id. Id 0 is reserved for padding and id 1 for unknown words, which the model will meet in production but not in training. Build the vocabulary from the training texts only, or the test set leaks into your preprocessing.

Reviews have different lengths, but tensors are rectangles. So the shorter ones are padded with id 0 up to the longest in the batch, and a mask records which positions are real. Without the mask, the padding would be averaged into the result and make the prediction depend on how long the other reviews in the batch happened to be.

Python Tokenizer, vocabulary and padding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The vocabulary has only 42 entries (2 special ones and 40 words, the most common being the, was, the comma and really). The padded batch of three reviews is 3 x 20: the first review has 20 tokens, the other two have 17, and their last three positions are zeros that the mask marks as padding. The unseen word "flabbergasting" becomes id 1, so unseen words degrade gracefully instead of crashing.

Step 2: the PyTorch model

The architecture is the simplest useful text classifier: embed, average the embeddings of the real tokens, and apply one linear layer. In PyTorch (read-only):

Python
import torch
from torch import nn
from torch.nn.utils.rnn import pad_sequence
from torch.utils.data import DataLoader, Dataset

class Reviews(Dataset):
    def __init__(self, texts, labels, vocab):
        self.ids = [torch.tensor([vocab.get(t, 1) for t in text.lower().split()]) for text in texts]
        self.labels = torch.tensor(labels, dtype=torch.float32)
    def __len__(self):
        return len(self.ids)
    def __getitem__(self, i):
        return self.ids[i], self.labels[i]

def collate(batch):
    ids, labels = zip(*batch)
    return pad_sequence(ids, batch_first=True, padding_value=0), torch.stack(labels)

class MeanPoolClassifier(nn.Module):
    def __init__(self, vocab_size, dim=8):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, dim, padding_idx=0)   # row 0 stays zero
        self.out = nn.Linear(dim, 1)

    def forward(self, ids):                                # ids: (batch, length)
        mask = (ids != 0).unsqueeze(-1)                    # (batch, length, 1)
        summed = (self.embedding(ids) * mask).sum(dim=1)   # padding adds nothing
        pooled = summed / mask.sum(dim=1)                  # mean over real tokens only
        return self.out(pooled).squeeze(-1)                # one logit per review

model = MeanPoolClassifier(len(vocab))
loss_fn = nn.BCEWithLogitsLoss()                           # sigmoid + binary cross-entropy, stable
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
loader = DataLoader(Reviews(train_texts, y_train, vocab), batch_size=64, shuffle=True, collate_fn=collate)

for epoch in range(20):
    for ids, labels in loader:
        optimizer.zero_grad()
        loss_fn(model(ids), labels).backward()
        optimizer.step()

A few points to notice. padding_idx=0 keeps the padding row of the table at zero and gives it no gradient, and the explicit mask makes the averaging correct. The model outputs a logit (a raw score), and BCEWithLogitsLoss applies the sigmoid internally, which is more stable than a separate Sigmoid layer followed by BCELoss (see Activation and Loss Functions). At prediction time, a positive logit means "positive review" (probability above 0.5). For many texts, nn.EmbeddingBag(vocab_size, dim, mode="mean") does the lookup and the mean in one memory-efficient step.

Step 3: the same model in NumPy, with gradients by hand

To see exactly what training does, the next editor trains this same architecture in plain NumPy. The forward pass is a table lookup, a masked mean and a dot product. The backward pass is short because every step is simple: for binary cross-entropy through a sigmoid, the gradient on the logit is (p - y) / n. That gradient flows into the linear weights (through the pooled vector) and into the embedding rows of the tokens that were present, each token getting an equal share of the mean. np.add.at scatter-adds those shares into the table, which is what nn.Embedding's backward does too, so a word that appears twice in a batch receives both contributions.

After training, an interesting property appears. Because the model is a mean of embeddings followed by a linear unit, the logit is the mean of per-word scores, and E @ w gives each word's score directly.

Python Mean-pooled embeddings with hand-written gradients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training converges quickly: the loss drops from 0.6932 (that is ln 2, a coin flip) to about 0.46 and stays there, and the test accuracy settles at about 78%. The learned word scores are sensible: disappointing, awful, annoying and dull are the most negative, and great, wonderful, amazing and lovely the most positive. And the two predictions at the bottom show the limit of the architecture. "the movie was great" is positive, as it should be, but "the movie was not great" is also classified as positive with 0.98 probability. The model averages word scores, so not is just one more word with one fixed score; it has no way to flip the meaning of the word next to it.

That is not a bug in the code, it is a property of bag-of-words style models: they ignore word order. The cure is a representation that sees combinations of words. Recurrent networks (see RNNs, LSTMs and GRUs) and attention (see Attention from Scratch) are the neural answers. The classical answer, which is easy to try right now, is to add n-grams: features for pairs and triples of adjacent words.

Step 4: add n-grams with scikit-learn

The next editor uses CountVectorizer, which tokenizes, builds the vocabulary and counts words for you (it returns a sparse matrix, just like a multi-hot encoding), and a logistic regression on top: the same "linear layer plus sigmoid" head as the neural model. It trains three versions that differ only in the features: single words, words plus pairs, and words plus pairs plus triples. A negated clause in our data has up to three tokens ("not really great"), which is why triples matter here.

Python Bag of words versus n-grams

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With single words, scikit-learn reaches 78.6%, practically the same as our NumPy model, which confirms that the weak spot is the representation and not the training. Adding word pairs lifts accuracy to 85.2%, and "not great" now gets a probability of positive of 0.23. Adding triples reaches 98.2%, because "not really great" is now a feature of its own. The price is the feature count: 39, then 247, then 1,141 features. On real text, n-gram vocabularies grow into the millions, which is one reason neural models with learned embeddings took over.

The last two printed lines come from the word-pair model: its strongest negative features are phrases such as not enjoyable and not great, and its strongest positive ones include not boring and not dull. The model discovered negation on its own, as a set of two-word patterns.

This is also how you should work on a real project. Start with the simplest baseline (a bag of words and logistic regression takes minutes), measure it, and only then build the neural model. If a neural model cannot beat the baseline, the problem is usually in the data or the pipeline, not in the architecture.

Extend the project

  • Hold out a validation set from the training texts and tune the embedding size (2, 8, 32) in the NumPy model.
  • Add L2 regularization (C=0.1 or 10) to the logistic regression and see how accuracy and feature weights change.
  • Make the generator harder: add "never" and "hardly" as extra negations, and watch which model breaks.
  • In PyTorch, replace mean pooling with an nn.LSTM over the embeddings and use its last hidden state (with pack_padded_sequence for the padding), or with a small nn.TransformerEncoder.
  • Report precision and recall, not just accuracy, once the classes are imbalanced.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the model need a mask when sequences are padded to the same length?

    Without a mask, the zero-padded positions would be included in the mean, so a review's prediction would depend on how much padding the batch happens to have.
  2. 2. The mean-pooled embedding model classifies "the movie was not great" as positive. Why?

    With mean pooling and a linear unit, the logit is just an average of fixed per-word scores. Recognizing negation requires seeing combinations of words, for example with n-grams, RNNs or attention.
  3. 3. Why does nn.BCEWithLogitsLoss usually beat a Sigmoid layer followed by nn.BCELoss?

    Fusing the sigmoid with the log of the loss avoids computing log(0) or overflow for extreme logits.
  4. 4. Where should the vocabulary be built from?

    Building the vocabulary from test data leaks information into preprocessing. Words not seen in training get the <unk> id in production.

Summary

  • A text classifier pipeline is: tokenize, vocabulary (with padding and unknown ids, built from training data only), pad with a mask, embed, pool, linear layer, sigmoid or softmax.
  • In PyTorch, nn.Embedding(padding_idx=0), a collate_fn with pad_sequence, a masked mean and BCEWithLogitsLoss implement it; nn.EmbeddingBag is a compact alternative.
  • With mean pooling, a word's contribution is one fixed score, so word order and negation are invisible.
  • N-gram features (or recurrent and attention models) bring local order back; on this dataset the accuracy rose from 79% to 85% to 98% as pairs and triples were added.
  • Always build a simple baseline first, and look at the model's most influential features to check that it learned something sensible.

Continue with Project: A Character-Level Language Model.