Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 63 ยท Word Meaning and Embeddings

Word Embeddings and Word2Vec

Understand dense word vectors and the skip-gram idea behind Word2Vec. Train a tiny embedding model in NumPy and try word analogies like king minus man.

A co-occurrence matrix captures word meaning, but it is huge and mostly zeros. A word embedding is the compact version: every word gets a short, dense vector of, typically, 50 to 300 real numbers, arranged so that words used in similar contexts end up with similar vectors. Instead of a 100,000-column row of counts, cat becomes something like [0.21, -0.43, 0.08, ...].

In 2013 a Google team led by Tomas Mikolov published Word2Vec, a fast and surprisingly simple way to learn such vectors from raw text, and embeddings became the standard first layer of nearly every neural NLP model. In this lesson you will see the central idea, the skip-gram model, and train a miniature version yourself in about 40 lines of NumPy.

Dense versus sparse

Sparse count vectors

Very long One column per vocabulary word.
Mostly zeros Most cells are empty.
Hand-built Counts and weights, no learning.
1 / 2

Because the dimensions are not tied to individual context words, two words can be close even if they never appeared near the same context word. Compression forces the model to find regularities, and that generalization is much of the benefit.

The skip-gram idea

Skip-gram trains a tiny prediction task: given a center word, predict the words around it. For the sentence "the king wore a crown" with a window of 2, the center word wore has to predict the, king, a and crown. The task itself is not the goal. The trained model is thrown away, and we keep the by-product: the weights that turned each center word into a vector.

The model has two weight matrices, each with one row per vocabulary word and d columns, where d is the embedding size:

  • W_in holds the embeddings: the row for the center word is its vector.
  • W_out holds one context vector per word.

To score how likely c is as a neighbor of center word w, take the dot product of W_in[w] and W_out[c]. Pass all scores through softmax to turn them into probabilities. Training adjusts both matrices with gradient descent so that real neighbors get high probability.

A related variant, CBOW (continuous bag of words), predicts the center word from the average of its context. Skip-gram works better for rare words, CBOW is faster. Both give similar quality embeddings.

The first example shows the plumbing on a single sentence: how training pairs are built, why an embedding is just a row lookup, and what an untrained model predicts.

Python Skip-gram pairs and the embedding lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A window of 2 turns five words into 14 pairs: edge words have fewer neighbors, which is why the and crown appear in only two pairs each as the center. Multiplying a one-hot vector by W_in simply selects a row, so real implementations skip the multiplication and index the matrix directly: the "embedding layer" in a neural network is exactly this lookup. Finally, an untrained model gives every context word roughly the same probability (about 0.2 for five words). Training will push wore and a above that for king, because they actually appear next to it.

Training and negative sampling

The loss for one pair is the negative log-probability of the true context word. Averaged over all pairs it starts near log(V), the cost of guessing uniformly, and falls as the model learns. Each step nudges the center word's embedding toward the context vectors of its real neighbors and away from the others.

There is a catch. The softmax normalizes over the whole vocabulary, so every single update costs time proportional to V: millions of operations for each of billions of pairs. Word2Vec avoids this with negative sampling: for each real pair, sample a handful of random words as "negatives" and train a small binary classifier that says "real neighbor" for the true pair and "not a neighbor" for the sampled ones. That makes each step cost a few dot products instead of V. Our toy vocabulary has 20 words, so we can afford the exact softmax.

Training a tiny model

The next example builds a toy corpus in which two things are visible in the context: whether a person is male or female (the word he or she appears nearby) and whether the person is royal (castle and crown) or ordinary (town and hat). It then trains an embedding for every word by gradient descent on the skip-gram loss, using nothing but NumPy matrix operations, and tests analogies.

Python Train a tiny Word2Vec in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts at 2.994, almost exactly log(20) for a vocabulary of 20 words, which is what uniform guessing costs. It falls to about 2.14 and flattens. It cannot reach zero, because the context of a word is genuinely uncertain: after king, either wore or is or walked could be nearby.

Look at what the vectors learned. The nearest neighbors of king are queen, boy and man, all nouns for people, with cosines above 0.95. The model has put the words for people close together, and, below, the differences between them are consistent.

Analogies

Mikolov's team noticed a famous property of embeddings trained on large corpora: directions in the space carry meaning. The vector difference king - man is roughly the "royalty" direction and woman - man the "female" direction, so

TEXT
king - man + woman  โ‰ˆ  queen

In the output, the nearest vector to king - man + woman, excluding the three input words, is queen (0.96), followed by boy and girl at 0.94. queen - woman + man gives king. The third test, girl - woman + man, is boy at 0.95, which is correct, but notice how close the others are: king is at 0.94.

Do not over-read analogies. They work here because the toy corpus was built to contain exactly these two regular differences. On real text they work for some relations (capital cities, verb tenses, gender) and fail on many others. The famous demonstrations are also selective: the test excludes the input words from the answer, and without that rule, the answer to king - man + woman is often just king. Treat analogies as a fun sanity check, not as evidence that the model "understands" anything.

Practical choices

  • Embedding size. 100 to 300 dimensions is typical. More dimensions capture finer distinctions but need more data.
  • Window size. As with count matrices, small windows capture syntax and large windows capture topics.
  • Corpus size. Word2Vec needs a lot of text: billions of words for the best general-purpose vectors. On a few hundred sentences, rare words get poor vectors. If you have little data, start from pre-trained vectors.
  • Libraries. In practice you would not write this yourself. Gensim's Word2Vec class trains with negative sampling in optimized C code (shown read-only below, since the site cannot install Gensim).
Python
from gensim.models import Word2Vec

model = Word2Vec(
    sentences=tokenized_sentences,  # a list of lists of words
    vector_size=100,
    window=5,
    min_count=5,        # ignore rare words
    sg=1,               # 1 = skip-gram, 0 = CBOW
    negative=5,         # negative samples per pair
    epochs=10,
)
print(model.wv.most_similar("king", topn=5))

What embeddings cannot do

One vector per word still means bank has a single blended vector. The vectors also reflect whatever is in the training text, including stereotypes, a topic we return to in the next lesson. And a word that never appeared in training has no vector at all; fastText, which builds vectors from character n-grams, addresses that. Later, contextual models give a different vector to each occurrence of a word.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the skip-gram model, what is the prediction task?

    Skip-gram takes a center word and learns to predict its neighbors. Predicting the center from the context is the CBOW variant.
  2. 2. After training, which part of the model do we keep as the word embeddings?

    Each row of W_in is the learned vector for one word. The prediction task is only a means of training those rows.
  3. 3. Why does Word2Vec use negative sampling?

    The full softmax costs time proportional to the vocabulary size. Negative sampling replaces it with a few binary classifications per pair.
  4. 4. What is an embedding layer in a neural network, mathematically?

    Multiplying a one-hot vector by a matrix selects one row, so frameworks implement the layer as an index into the matrix.
  5. 5. A trained model returns "queen" for king - man + woman. What is the best interpretation?

    It shows that a particular relation is encoded as a consistent offset in this embedding space, which is interesting but not proof of understanding, and it does not hold for every relation.

Summary

  • A word embedding is a short, dense, learned vector for each word; similar words get similar vectors.
  • Skip-gram learns embeddings by predicting the neighbors of a center word; CBOW predicts the center from its context.
  • The embeddings are the rows of the input weight matrix; an embedding layer is a row lookup.
  • Negative sampling makes training fast by replacing the full softmax with a few binary decisions.
  • Relations such as male-to-female can appear as consistent vector offsets, which is what analogies test, but they are not guaranteed.
  • Embeddings give one vector per word, inherit biases from their data, and need lots of text to train well.

Next, see another way to get dense vectors and how to use ready-made ones in Embeddings from SVD and Using Pre-trained Vectors.