Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 45 ยท Sequences and Transformers

Embeddings and Text Data

Prepare text for PyTorch: tokenization, vocabularies, padding and masks, nn.Embedding lookups, mean pooling and a text classifier skeleton with live NumPy stand-ins.

Neural networks work on numbers, but text arrives as strings. Turning words into something a model can learn from takes a small chain of steps: split the text into tokens, map each token to an integer id using a vocabulary, pad the integer sequences into rectangular batches, and finally look each id up in an embedding table, which gives every token a learnable vector. The vectors start random and are trained along with the rest of the model, so tokens used in similar ways end up with similar vectors.

This lesson walks through that pipeline and ends with a skeleton of a text classifier. PyTorch cannot run on this site, so the PyTorch code is read-only; the live editors rebuild the pipeline with NumPy (a vocabulary, padding, an embedding lookup, and even a tiny classifier whose embeddings are trained by hand). They are stand-ins for nn.Embedding and friends, built from the same arithmetic. You may want to refresh Datasets and DataLoaders first, because batching text relies on a custom collate_fn.

From text to tensors

Tokenize Split text into tokens.
Vocabulary Give every token an integer id.
Pad Make all sequences equally long.
Embed Look up one vector per id.

Each step is simple. The order is what matters.

1 / 2

Tokenization and the vocabulary

The simplest tokenizer lowercases the text and splits on whitespace and punctuation. Production systems use subword tokenizers (byte-pair encoding, WordPiece, SentencePiece) that break rare words into reusable pieces, so unknown words become rarer; libraries such as tokenizers or transformers provide them. For the idea, a word-level tokenizer is enough.

A vocabulary maps each distinct token to an integer. You build it from the training data only, usually keeping the most frequent tokens, and reserve ids for two special tokens: <pad> (filler, id 0 by convention) and <unk> (any token that is not in the vocabulary, for example one that appears only in test data).

Python
import re
from collections import Counter

def tokenize(text):
    return re.findall(r"[a-z0-9']+", text.lower())

train_texts = ["The movie was great!", "Great acting, terrible plot.", "I loved it"]
counts = Counter(tok for text in train_texts for tok in tokenize(text))

itos = ["<pad>", "<unk>"] + [tok for tok, n in counts.most_common() if n >= 1]
stoi = {tok: i for i, tok in enumerate(itos)}

def encode(text):
    return [stoi.get(tok, stoi["<unk>"]) for tok in tokenize(text)]

print(encode("The plot was great"))   # a list of ids, one per token

Two traps: building the vocabulary on the whole dataset (including validation and test data) leaks information, and forgetting <unk> makes the code crash on the first unseen word.

Embeddings are lookup tables

nn.Embedding(num_embeddings, embedding_dim) is a trainable matrix with one row per vocabulary entry. Feeding it integer ids returns the matching rows:

Python
import torch
from torch import nn

emb = nn.Embedding(num_embeddings=len(itos), embedding_dim=8, padding_idx=0)
ids = torch.tensor([[2, 5, 3, 0],
                    [4, 1, 0, 0]])         # (batch=2, seq_len=4), 0 is <pad>

vectors = emb(ids)
print(vectors.shape)                       # torch.Size([2, 4, 8])
print(emb.weight.shape)                    # torch.Size([vocab_size, 8])
print(vectors[0, 3])                       # all zeros: padding_idx rows start at 0 and get no gradient

The output has one extra dimension: (batch, seq_len) ids become (batch, seq_len, embedding_dim) vectors, which is exactly what an RNN or a Transformer expects as input (see RNNs, LSTMs and GRUs). Mathematically, an embedding lookup is the same as multiplying a one-hot vector by the weight matrix, only without ever building the huge one-hot vector, so it is fast and uses memory proportional to the number of tokens in the batch.

padding_idx=0 tells PyTorch that row 0 is filler: it is initialized to zeros and never receives gradient updates. The input must be integer tensors (torch.long); passing floats raises an error. An out-of-range id raises an IndexError, which is the usual symptom of a vocabulary mismatch between training and inference.

You can also start from pretrained vectors (GloVe, fastText and similar) with nn.Embedding.from_pretrained(matrix, freeze=True), which is a form of the transfer learning idea from Transfer Learning and Fine-Tuning.

The first editor builds a vocabulary, encodes and pads three sentences, and shows that an embedding lookup equals a one-hot matrix product.

Python Vocabulary, padding and an embedding lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Masks and pooling

A batch of padded sequences has (batch, seq_len, dim) vectors, but a classifier needs one vector per text. The simplest way to get it is pooling: average (or take the maximum of) the token vectors. Averaging must ignore padding, otherwise short texts get diluted by zeros. With a mask of shape (batch, seq_len) that is True for real tokens:

Python
mask = (ids != 0).unsqueeze(-1)                  # (batch, seq_len, 1)
summed = (vectors * mask).sum(dim=1)             # padded positions contribute 0
counts = mask.sum(dim=1).clamp(min=1)            # real tokens per text
mean_pooled = summed / counts                    # (batch, embedding_dim)

Masks matter in three more places: attention scores must not look at padded positions (see Attention from Scratch), recurrent layers should skip padding (packing), and the loss must ignore padded targets (the ignore_index argument of CrossEntropyLoss).

PyTorch also offers nn.EmbeddingBag, which embeds and averages (or sums) variable-length bags of ids in one call using offsets, with no padding at all. It is a good fit for bag-of-words style models.

A text classifier skeleton

Putting it all together, here is the whole pipeline for sentiment classification: collate_fn pads each batch, the model embeds, pools and classifies.

Python
from torch.nn.utils.rnn import pad_sequence
from torch.utils.data import DataLoader

def collate(batch):
    texts, labels = zip(*batch)
    ids = pad_sequence([torch.tensor(encode(t)) for t in texts],
                       batch_first=True, padding_value=0)
    return ids, torch.tensor(labels, dtype=torch.float32)

class TextClassifier(nn.Module):
    def __init__(self, vocab_size, dim=64):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, dim, padding_idx=0)
        self.head = nn.Linear(dim, 1)

    def forward(self, ids):
        mask = (ids != 0).unsqueeze(-1)
        pooled = (self.embedding(ids) * mask).sum(1) / mask.sum(1).clamp(min=1)
        return self.head(pooled).squeeze(-1)             # one logit per text

model = TextClassifier(len(itos))
loss_fn = nn.BCEWithLogitsLoss()                         # binary labels, raw logits
loader = DataLoader(list(zip(train_texts, [1, 0, 1])), batch_size=2,
                    shuffle=True, collate_fn=collate)

Training then follows the usual loop from The Complete Training Loop. The model is tiny but surprisingly strong for topic or sentiment tasks, because averaging word vectors captures which words are present; it ignores word order, which is where recurrent and attention models come in. The loss is explained in Activation Functions and Loss Functions.

Training embeddings by hand

Where do good vectors come from? From the same gradient descent as everywhere else. The next editor trains the model above in NumPy on a small synthetic set of reviews: each text is a few words drawn from positive, negative and neutral lists. The code computes the gradients of the embedding table by hand (the job loss.backward() does for you) and updates only the rows of the words that appeared. Afterwards, words with the same sentiment end up closer to each other than to words of the opposite sentiment.

Python A tiny text classifier with trained embeddings (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real embeddings are trained the same way but on millions of texts, with dimensions of 100 to 4,096 or more, and they capture much richer structure than positive versus negative.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does nn.Embedding(1000, 32) contain?

    An embedding layer is a lookup table with one row per id (num_embeddings) and embedding_dim columns. It is trained like any other weight.
  2. 2. Input ids have shape (8, 20) and are passed through nn.Embedding(5000, 64). What is the output shape?

    Each of the 8 x 20 ids is replaced by its 64-number vector, so one dimension is added at the end.
  3. 3. Why average token vectors with a mask instead of a plain mean over the sequence axis?

    Padded positions are not real tokens. Summing with the mask and dividing by the number of real tokens gives the true average.
  4. 4. A word appears in the test set but never in the training data. How should the model see it?

    The vocabulary is fixed from the training data. Unknown words map to <unk>; padding must stay reserved for filler so masks remain correct.

Summary

  • Text becomes tensors in four steps: tokenize, map to ids with a vocabulary (with <pad> and <unk>), pad, then embed.
  • nn.Embedding(vocab, dim, padding_idx=0) is a trainable lookup table that turns (batch, seq) ids into (batch, seq, dim) vectors; it equals a one-hot matrix product without the memory cost.
  • Build the vocabulary from training data only, and use masks so padding never counts in pooling, attention or the loss.
  • Mean-pooled embeddings plus a linear head make a small, strong text classifier that ignores word order.
  • Embeddings learn from gradient descent, and can also start from pretrained vectors.

Next, learn how models decide which tokens matter for each other in Attention from Scratch.