Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 63 ยท Word Meaning and Embeddings

Embeddings from SVD and Using Pre-trained Vectors

Compress a PPMI matrix into dense word vectors with SVD, load GloVe and fastText files, and see how embeddings absorb bias from the text they learn from.

Word2Vec learns vectors by training a small network. There is an older, more mathematical route to the same destination: take the PPMI co-occurrence matrix and compress it with a matrix factorization called singular value decomposition (SVD). The result is a dense vector per word, built with a single call to linear algebra and no training loop.

This lesson shows how that works, how to read the pre-trained vectors that other people have already computed from billions of words (GloVe and fastText), and, importantly, why such vectors carry the biases of the text they came from.

SVD in one picture

SVD splits a matrix P into three parts: P = U ยท diag(S) ยท Vt. The numbers in S, the singular values, are sorted from large to small and tell you how much of the matrix's structure each "direction" carries. If you keep only the first k directions and drop the rest, you get the best possible rank-k approximation of P (in the least-squares sense). The rows of U[:, :k] * S[:k] are then k-dimensional vectors, one per word.

From counts to dense vectors

Count Build the co-occurrence matrix.
Weight Turn counts into PPMI.
Compress Keep the top k singular directions.
Use Each row is a word vector.
1 / 2

Why does cutting dimensions help? The top directions capture the broad, repeated patterns (animals appear in these kinds of sentences), while the tail captures accidents of a particular corpus. Dropping the tail acts like noise removal, and it makes words that share patterns close even when they share few exact context words. This idea is also known as latent semantic analysis (LSA) when applied to a document-term matrix.

Building SVD embeddings

The example rebuilds the toy corpus from the previous lessons (animals, foods and vehicles), computes PPMI, and applies SVD. It first prints how the choice of k changes similarity, then compares our NumPy result with scikit-learn's TruncatedSVD, which is the class you would use on a large sparse matrix (it avoids computing the full decomposition).

Python PPMI plus SVD, and the effect of k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first block is the lesson about k. With a single dimension (k=1) every word has cosine 1.0 with every other word, because all non-negative PPMI rows share the dominant direction: no information survives. With k=2 animals and foods are still partly blended (cat~bread is 0.87), and at k=3 the topics separate: cat~dog stays near 1 while cat~bread drops to 0.22 and cat~car to about zero. Pushing k up to 20 slowly lowers cat~dog to 0.75, since the extra directions start modeling the quirks of individual words. This corpus has three topics, so three dimensions already tell them apart; real language needs hundreds.

The NumPy result and TruncatedSVD agree (0.99 and 0.2) even though the individual coordinates can differ in sign, because cosines do not depend on the signs of axes. The matrix shrank from 51 columns to 5, and the nearest neighbors are what you would hope: cat is next to horse, rabbit and dog, and bread next to soup, apple and rice.

SVD on PPMI is a strong baseline that is easy to reproduce. Word2Vec and GloVe usually do a little better on big corpora, and a classic paper showed that Word2Vec's skip-gram with negative sampling is implicitly factorizing a shifted PMI matrix, so the two routes are closer relatives than they look.

Using pre-trained vectors

Training good embeddings needs billions of words and a lot of compute. Fortunately, you rarely have to: researchers publish the finished vectors. The three families you will meet:

  • Word2Vec: Google's vectors trained on news text, 300 dimensions, distributed in a binary format.
  • GloVe (Stanford): trained on global co-occurrence counts by fitting each pair's dot product to the logarithm of its count. The glove.6B files cover 400,000 words, trained on 6 billion tokens, in 50, 100, 200 and 300 dimensions.
  • fastText (Meta): like skip-gram, but each word is also represented by its character n-grams, so it can build a vector for a word it never saw (unfriendliest, a typo, a new brand name) from the pieces it recognizes.

GloVe's text format is as simple as formats get: one line per word, the word first, then its numbers separated by spaces. fastText .vec files are the same but start with a header line, <number of words> <dimensions>. Here is how real files are used (read-only, because the site cannot download hundreds of megabytes):

Python
# A real GloVe file, loaded with Gensim
from gensim.models import KeyedVectors
wv = KeyedVectors.load_word2vec_format("glove.6B.100d.txt", binary=False, no_header=True)
print(wv.most_similar("python", topn=5))
print(wv.most_similar(positive=["king", "woman"], negative=["man"], topn=3))
print(wv.similarity("coffee", "tea"))

The next example does the same job on a tiny hand-made file held in memory, with the loader written from scratch. The six vectors are invented for illustration, not real GloVe numbers.

Python Reading a GloVe-format file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three practical points show up in the output. First, zebra is out of vocabulary: a plain GloVe or Word2Vec table has no vector for a word it did not see, which is exactly the gap fastText closes. Second, the analogy lands on queen because we built the numbers that way. Third, averaging word vectors gives a vector for a phrase, which is the starting point of the next lesson.

Practical advice for real projects: lowercase and tokenize your text the same way the vectors' authors did (GloVe's files are lowercased), choose a dimension that fits your memory budget (a 400,000 by 300 table of 32-bit floats is about 480 MB), and decide in advance what to do with out-of-vocabulary words: skip them, use a zero vector, or use fastText.

Bias in embeddings

Embeddings are trained on human-written text, and human-written text is not neutral. Studies of real GloVe and Word2Vec vectors have repeatedly found that stereotypes are encoded in them: occupations such as nurse and engineer lie on opposite sides of the "he versus she" direction, and some names and groups pick up positive or negative associations from how they are written about. Whatever model you build on top inherits this.

You can see the mechanism with a toy corpus in which the pronoun use is deliberately skewed: engineers and pilots are "he" in 90% of sentences, nurses in only 10%. The example builds PPMI plus SVD embeddings with three dimensions, then measures each job's bias as its cosine with he minus its cosine with she. It finally removes the he-she direction from the job vectors. This corpus is invented to show the effect. It says nothing about real people.

Python Measuring and removing a gender direction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bias scores follow the data exactly: engineer and pilot are at about +0.45, nurse at -0.47, and writer, whose sentences are split evenly, is close to zero (0.11). Nothing in the algorithm "decided" this. It is simply what the counts say, compressed faithfully.

Removing the direction pushes every job's score to 0.00, which looks like a fix. Read the last two lines before celebrating. After the removal, nurse and engineer have cosine 1.0, because the only thing that distinguished them in this corpus was gender, so the removal erased all the difference. On real text the situation is subtler, and research on projection-based "debiasing" found that it mostly hides the bias on one axis: stereotyped words still cluster together, and the groups can be recovered from the remaining geometry. There is no single trick that makes embeddings fair.

Bias in embeddings matters whenever vectors feed a decision about people: ranking rรฉsumรฉs, moderating comments, scoring loan applications. Measure it on your own data, document known limits, and keep a human in the loop. We return to this in Evaluation, Bias and Responsible NLP.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does keeping only the top k singular directions of a PPMI matrix give you?

    Rows of U[:, :k] times the top k singular values are dense k-dimensional word vectors, the best rank-k approximation of the matrix.
  2. 2. In the k experiment, why did every word have cosine 1.0 with every other word when k was 1?

    PPMI values are non-negative, so the first singular direction is shared by every row. One number per word cannot separate topics.
  3. 3. What does fastText do that a plain GloVe table cannot?

    fastText represents a word as the sum of its character n-gram vectors, so unseen words and typos still receive a sensible vector.
  4. 4. A GloVe .txt file is loaded and the line "cat 0.1 -0.3 0.7" is parsed. What does it contain?

    The format is one word per line followed by the vector's numbers. fastText .vec files add a header line with the word count and dimensions.
  5. 5. An embedding places "nurse" closer to "she" and "engineer" closer to "he". Where does this come from?

    Embeddings summarize how words are used. If the text pairs jobs with pronouns unevenly, the vectors encode that association.

Summary

  • SVD compresses a PPMI matrix into dense vectors: keep the top k singular directions and use the rows of U * S.
  • The value of k is a trade-off; too small blurs topics, too large brings back corpus noise.
  • Scikit-learn's TruncatedSVD does this efficiently for large sparse matrices; the idea is also called latent semantic analysis.
  • Pre-trained vectors (Word2Vec, GloVe, fastText) save you from training on billions of words; GloVe files are plain text with one word and its numbers per line.
  • Plain word tables fail on out-of-vocabulary words; fastText builds vectors from character n-grams.
  • Embeddings inherit the stereotypes of their training text; removing one direction hides some bias but does not make vectors fair.

Next, learn how to turn whole sentences and documents into vectors in Sentence and Document Embeddings.