Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 73 ยท Sequences, Attention and Transformers

Embeddings

Learn embeddings as trainable lookup tables: train word vectors on a toy corpus in NumPy, compare them with cosine similarity, and embed categories.

Neural networks work with numbers, but words, product IDs, user IDs and weekdays are labels. The naive way to feed a label to a network is one-hot encoding: a vector as long as the vocabulary, all zeros except a single 1. It works, but it has two flaws. It is huge (a 50,000-word vocabulary means 50,000 numbers per word) and it is blind: in a one-hot world, "cat" is exactly as different from "dog" as it is from "carburetor".

An embedding replaces each label with a short, dense vector of real numbers (typically 8 to 1,024 of them) that the network learns. Items that behave similarly end up with similar vectors, so the model can generalize from what it learned about one item to its neighbors. Embeddings are the entry point of every language model, the heart of recommendation systems, and a standard way to handle categorical columns in tabular data. In this lesson you will build one from scratch.

One-hot versus embedding

One-hot Long, sparse, and every word is equally far from every other.
Embedding Short, dense, and similar words get nearby vectors.

An embedding is a lookup table whose rows are learned parameters.

1 / 2

An embedding is a lookup table

An embedding layer holds a matrix E of shape (vocab_size, dim). To embed the token with ID 7, you take row 7 of E. That is all. Mathematically it is the same as multiplying the one-hot vector by E, but indexing is far cheaper than multiplying by 50,000 zeros.

During training, the gradient arriving at the output is added to the rows that were used and to nothing else. A token that appears twice in the input receives both gradients, added together (np.add.at does this in NumPy, as you used for the input weights in Backpropagation Through Time). Rows of tokens that do not appear in the batch stay untouched.

Python An embedding layer is a table lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All five rows get a gradient here because all five words appear in the sentence. The mean of the word vectors is the simplest way to turn a variable-length sentence into one fixed-size vector, a trick you will use in the sentiment project. The table of a real vocabulary is big: 15 million numbers for 50,000 words and 300 dimensions, which is why embedding tables often hold most of a model's parameters.

Measuring similarity: the cosine

How do you tell whether two embeddings are "close"? The common answer is cosine similarity, the cosine of the angle between the vectors:

cos(a, b) = (a . b) / (|a| * |b|)

It is 1 when the vectors point in the same direction, 0 when they are perpendicular, and -1 when they point in opposite directions. Unlike Euclidean distance, it ignores the length of the vectors. That matters because the length of a learned vector often reflects how frequent the word is, not what it means.

Python Cosine similarity ignores length

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By cosine, cat and cat_big are identical (1.0) even though they are 20 units apart in Euclidean distance, and dog is much closer to cat than car is. If you compare many vectors, normalize each to unit length once and then a plain dot product is the cosine.

Learning word vectors from context

Where do good vectors come from? From a simple idea in linguistics: a word is known by the company it keeps. Words that appear in similar contexts, such as "cat" and "dog" (both are followed by "runs" and preceded by "the"), should get similar vectors.

The skip-gram model turns that idea into a prediction task. Given a center word, predict a word that appears near it. The network is tiny: look up the center word's vector E[w], multiply by an output matrix U of shape (dim, vocab) to get a score for every possible context word, and apply softmax with cross-entropy loss. To make the prediction work, words used in similar contexts must have similar vectors, and that is how the meaning ends up in the table.

We train on a toy corpus of about 50 short template sentences about animals, foods, people and royalty. Real systems sample billions of word pairs. Here the corpus is small enough that we can count how often each pair occurs within two positions of each other and minimize the average skip-gram loss over all pairs at once, which is the same objective in a deterministic, fast form.

Python Training skip-gram word vectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss drops from 3.93 to about 2.14 and then stops, because the next word is never perfectly predictable, so this is the floor for this corpus. The neighbors are what you hope for. Every animal's nearest neighbors are other animals, with a cosine of 1.00 in this toy case: the five animals appear in exactly the same sentence templates, so they have the same contexts and therefore the same vectors. The foods cluster the same way. woman is closest to girl (1.00), and king has queen, man and boy nearby, because they share patterns such as "the ... said he".

The analogy lines compute king - man + woman and look for the nearest vector (excluding the three input words). The nearest is queen at 0.98. Likewise queen - woman + man gives king, and boy - man + woman gives girl. The reason is that the corpus ties the word pairs together through a consistent gender direction (the pronouns "he" and "she"). The 2-D picture, made with the principal components of the 10-D table, shows the animals, the foods and the people each forming their own group. Remember that this is a toy: with only a handful of contexts per word, ties such as the animals' cosine of 1.00 are exactly what you would expect.

Real embeddings (word2vec, GloVe, or the embedding layers inside language models) are trained on billions of words, so their neighborhoods and analogies are far richer, but also noisier and biased by their training text. Analogy arithmetic works best on clean, frequent relationships, as in our toy corpus.

Embeddings for categorical features

Words are not the only labels. A tabular dataset may have columns like weekday, country, product_id or user_id. One-hot encoding a column with a million users is impractical, and it gives the model no way to say "these two users are alike". The fix is the same: one embedding row per category, trained together with the rest of the network. A common rule of thumb for the dimension is a small fraction of the category count, such as min(50, categories // 2) or its fourth root; it is a hyperparameter to tune.

In the example below, a network predicts a store's daily sales from the weekday and the temperature. Sales are low from Monday to Thursday, rise on Friday and peak at the weekend. The weekday goes through a 2-dimensional embedding; the temperature is passed in as a plain number. Nothing tells the network that days are related. It has to discover it.

Python Learning a weekday embedding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error falls from 1.31 to 0.014 (the targets were standardized, so 1.0 is the variance of the data). Look at the last table: the distance from Monday grows with the difference in sales. Wednesday and Thursday, with similar sales, sit very close to Monday (0.17 and 0.27), Friday is farther (0.86), and Saturday and Sunday, the busy days, are far away (1.44 and 1.68). Tuesday (0.56) is a little off the pattern, which is normal for a 2-D embedding learned from noisy data. Nobody told the network about weekends; the training signal pulled the days with similar behavior together.

Practical notes

  • Initialization and size. Embedding tables start with small random values and are trained like any other weights. Use a larger dimension when you have lots of data and a rich vocabulary.
  • Pretrained embeddings. You can start from vectors trained on a huge corpus and either freeze them or fine-tune them, the same idea as in Transfer Learning and Fine-Tuning.
  • Unknown items. Reserve a row for an "unknown" token, and optionally for padding (whose row is ignored with a mask, see Working With Sequences).
  • Beyond words. Sub-word tokens, users, items, graph nodes, images and whole sentences are all embedded in the same way. Positions in a sequence get embeddings too, which you will meet in Self-Attention and Multi-Head Attention.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is an embedding layer, computationally?

    An embedding layer indexes a (vocab_size, dim) matrix. Multiplying a one-hot vector by it would give the same row, but indexing is much cheaper.
  2. 2. Why is cosine similarity preferred to Euclidean distance for comparing embeddings?

    Two vectors in the same direction have cosine 1 regardless of length. Length is often influenced by how frequent a word is rather than what it means.
  3. 3. In the skip-gram setup, what is the model trained to do?

    Skip-gram predicts nearby context words from the center word's vector. Words that share contexts must get similar vectors to make good predictions.
  4. 4. During backpropagation through an embedding layer, which rows of the table receive a gradient in one batch?

    The gradient is scattered back to the rows that were looked up; repeated tokens add their contributions. Other rows are untouched.
  5. 5. Which is a good use of embeddings outside of text?

    Embeddings give each category a learned dense vector, which is far more practical than a million-wide one-hot column and lets similar categories share statistical strength.

Summary

  • An embedding is a trainable lookup table that maps IDs to dense vectors; it equals a one-hot vector times a matrix, computed by indexing.
  • Gradients flow only to the rows that were used, and repeated tokens add up.
  • Cosine similarity compares directions and ignores vector length.
  • Skip-gram learns word vectors by predicting context words, so words with similar contexts get similar vectors; simple analogies such as king - man + woman can work on clean data.
  • The same mechanism embeds categorical columns, users, items and positions, and the embedding dimension is a hyperparameter.
  • Embeddings are the first layer of language models, and the vectors they produce feed the attention mechanism in the next lesson.

Next: Seq2Seq, Teacher Forcing and Beam Search.