Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 63 ยท Neural NLP and Transformers

Neural Networks for Text

Build a small neural text classifier from scratch in NumPy: embedding layer, mean pooling, hidden layer, softmax, backpropagation and a training loop.

Until now, our classifiers took fixed features, such as TF-IDF weights, and learned only a final decision on top. A neural network for text learns the representation itself. The words go in as integer ids, the first layer turns each id into a vector, and every layer after that is trained together, so the word vectors end up shaped by the task. A sentiment classifier learns vectors in which brilliant and wonderful sit together and far from dreadful, because that is what lowers its loss.

This lesson builds the smallest useful neural text model from scratch in NumPy, so that no step is hidden: an embedding layer, mean pooling, one hidden layer and a softmax output. You will see the shapes at every step, write the training loop with backpropagation, and then look at what the model can and cannot do. The same structure, with bigger parts, sits inside every model in the following lessons.

The architecture

A tiny text classifier

Token ids Each word becomes an integer.
Embedding layer A table lookup gives a vector per word.
Mean pooling Average the vectors into one.
Hidden layer A weight matrix plus ReLU.

The output layer then turns the hidden values into class probabilities.

1 / 2

Here is each piece in turn.

  • Token ids. The vocabulary maps every word to a row number. Special entries are customary: <pad> (id 0) fills short sentences so a batch is a rectangle, and <unk> stands for any word not in the vocabulary.
  • Embedding layer. A matrix E with one row per word and d columns. Looking up the rows for a sentence's ids gives an array of shape (words, d). As you saw in the Word2Vec lesson, this is the same as multiplying one-hot vectors by E, just faster.
  • Mean pooling. Sentences differ in length, but the next layer needs a fixed-size input, so we average the word vectors, ignoring padding. This is the "average word vectors" idea from the previous lesson, except now the vectors are learned.
  • Hidden layer. h = ReLU(pooled @ W1 + b1). ReLU replaces negative numbers with 0, which lets the network represent more than straight lines.
  • Output layer. logits = h @ W2 + b2, then softmax to get one probability per class.

The first example runs a single sentence through an untrained network and prints every shape, so you can follow the arithmetic.

Python One forward pass, step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four words become ids [2, 3, 4, 5], then a (4, 4) array (four words by four dimensions), then a single vector of four numbers, then three hidden values (two of them zeroed by ReLU), and finally two probabilities of about 0.5 each: an untrained network is just guessing. Notice the parameter count. Of 59 numbers, 36 are embeddings, and in real models the share is far bigger: a 50,000-word vocabulary with 300 dimensions is 15 million parameters in the embedding table alone. The last line shows what happens to a word the model has never seen, film: it is mapped to id 1, the <unk> row, and the model has no idea what it meant.

Training: forward, loss, backward, update

To train, we repeat four steps until the loss stops falling:

flowchart LR
    F["Forward pass<br/>predict"] --> L["Loss<br/>how wrong?"]
    L --> B["Backward pass<br/>gradients"]
    B --> U["Update<br/>weights -= lr * gradient"]
    U --> F

Each loop over the data is an epoch; a model typically needs tens to hundreds of them.

The loss here is cross-entropy, the negative log of the probability the model gave to the correct class: confident and right gives a loss near 0, confident and wrong gives a large loss. The backward pass applies the chain rule layer by layer to find how much each weight contributed to the loss. For a softmax output, the gradient on the logits has a beautifully simple form: the predicted probabilities minus a one-hot vector for the true class. From there the gradient flows back through the weights, through the ReLU (blocked wherever the hidden value was zero), through the average (each word receives an equal share) and into the rows of the embedding table. Only the rows of words that appeared in the batch change.

The second example trains this network end to end. To have data of our own, it builds 300 short reviews from templates (positive or negative adjectives in sentences such as "what a lovely movie") and keeps another 100 for testing. All the code is NumPy; the gradient steps match the description above.

Python Train a neural text classifier in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the log. At epoch 0 the loss is 0.704, about log(2) for two classes, and accuracy is near chance. By epoch 80 the loss has dropped to 0.219 and the model classifies the training set perfectly; by epoch 200 the loss is 0.004. The test accuracy follows closely (0.99, then 1.00). In this toy setting that is no surprise: the test sentences use the same templates and the same 29 words, so there is nothing new to generalize to. Do not take 100% as a promise about real reviews, where vocabulary, style and sarcasm vary far more.

The four predictions at the end tell the real story:

  • the movie was brilliant gets 1.00 and what a dreadful ending gets 0.00. The adjectives were seen in training, and the model learned their signs.
  • the plot was dull and the acting was great gets 0.33. One negative and one positive word average out, and the network leans slightly negative; mean pooling cannot weigh which clause matters more.
  • the movie was stellar gets 0.35. stellar was never seen, so it maps to <unk>, whose row was never trained: the model has no idea whether it is good, and the result is just whatever the other words and the untrained <unk> vector add up to.

That last point is the real weakness of learning embeddings from a small dataset: they are only as good as the vocabulary the training data covers. The standard remedies are to start from pre-trained vectors, which already know that stellar is close to brilliant, or to use subword tokenization so that unseen words are built from known pieces.

The same thing in a framework

Nobody writes backpropagation by hand for real work. Frameworks compute gradients automatically. The model above in PyTorch is a few lines (read-only: PyTorch is not available in the browser):

Python
import torch
from torch import nn

class TextClassifier(nn.Module):
    def __init__(self, vocab_size, dim=8, hidden=8, classes=2):
        super().__init__()
        self.embed = nn.EmbeddingBag(vocab_size, dim, mode="mean", padding_idx=0)
        self.net = nn.Sequential(nn.Linear(dim, hidden), nn.ReLU(), nn.Linear(hidden, classes))

    def forward(self, ids):          # ids: (batch, words), 0 is padding
        return self.net(self.embed(ids))

model = TextClassifier(vocab_size=29)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)

for epoch in range(200):
    optimizer.zero_grad()
    loss = loss_fn(model(X_train), y_train)   # forward + loss
    loss.backward()                            # the backward pass, automatic
    optimizer.step()                           # weights -= lr * gradient

nn.EmbeddingBag with mode="mean" is the embedding lookup plus mean pooling in one operation, and loss.backward() replaces every gradient line in our loop. Everything else is the same.

Practical notes

  • Learning rate. Too large and the loss jumps around or explodes; too small and training crawls. 0.5 suits this tiny, smooth problem; real networks usually use smaller rates and optimizers such as Adam.
  • Overfitting. A model with many parameters can memorize the training set. Watch the test (validation) accuracy, and use dropout, weight decay or early stopping when the gap grows.
  • Batches. Real datasets are processed in mini-batches rather than all at once, which fits in memory and adds helpful noise.
  • Versus linear models. On small datasets with plenty of overlapping words, logistic regression on TF-IDF is often just as accurate, faster to train and easier to interpret. Neural networks pay off with more data, when word order and context matter, and when embeddings can be shared across tasks.
  • Order still matters. Mean pooling throws away word order, exactly as in the last lesson. The next lessons fix this: recurrent networks read the words one at a time, and transformers let every word look at every other.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the embedding layer of a neural text model do?

    The layer is a matrix with one row per vocabulary word. Looking up the rows for a sentence's ids gives its word vectors, and those rows are learned along with the rest of the network.
  2. 2. Why does the model use mean pooling after the embedding layer?

    Sentences have different lengths, but the next layer expects a fixed-size input. The average gives one vector per sentence, at the cost of ignoring word order.
  3. 3. For a softmax output with cross-entropy loss, what is the gradient with respect to the logits?

    This simple form is what the backward pass starts from. It shrinks toward zero as the correct class gets a probability near 1.
  4. 4. A trained model sees the word "stellar", which was not in its training vocabulary. What happens?

    Out-of-vocabulary words map to the shared unknown token. Pre-trained vectors and subword tokenization are the usual ways to handle them better.
  5. 5. In the training log, test accuracy reached 1.00. Why should you not read this as a promise for real reviews?

    The toy test set contains nothing new, so it does not measure generalization to different phrasing, new words or sarcasm.

Summary

  • A neural text model learns its own features: ids go in, an embedding layer looks up vectors, pooling makes one vector, layers and softmax give probabilities.
  • The embedding table is usually most of the parameters; its rows are trained for the task.
  • Training repeats forward pass, loss, backward pass and update; the gradient on the logits is the probabilities minus a one-hot vector.
  • Words unseen in training map to <unk> and are not understood; pre-trained vectors or subwords help.
  • Mean pooling ignores word order and cannot weigh conflicting clauses.
  • Frameworks such as PyTorch compute the backward pass automatically; the structure is the same as the NumPy version.

Next, teach the network to read words in order with Recurrent Networks and LSTMs.