Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 63 ยท Word Meaning and Embeddings

Distributional Semantics

Learn how a word's meaning can be read from the company it keeps. Build co-occurrence matrices, apply PPMI weighting and compare word vectors with cosine.

So far, words in this course have been symbols. In a bag of words, car and automobile are two unrelated columns, and the only thing that tells you two documents are alike is that they share exact word forms. That is a hard ceiling. To go further, a program needs some notion of what a word means, or at least which words behave alike.

Linguists gave a surprisingly practical answer in the 1950s: "You shall know a word by the company it keeps." If two words show up in similar contexts, they probably have similar meanings. This is the distributional hypothesis, and it is the foundation of every word embedding and, indirectly, of today's large language models. In this lesson you will turn it into numbers with the simplest possible tools: counting and cosine similarity.

The distributional idea

Think about the sentence "We fed the ___ this morning." Cat, dog and horse all fit. Bread does not. You never needed a dictionary to know that: the context narrows the possibilities. Turn that around and you get a recipe. Collect every context a word appears in, and use that collection as the word's representation.

Meaning from context

Look around Collect the words near each target word.
Count Store the counts in a matrix.
Compare Similar rows mean similar words.

No dictionary and no labels are needed, only raw text.

1 / 2

The word-context matrix

A co-occurrence matrix has one row per word and one column per context word. The cell at row w and column c counts how often c appears within a context window around w. A window of 2 means "up to two words to the left and two to the right".

To keep the demo small and fully reproducible, the next example generates a toy corpus of 300 sentences from templates: some about animals, some about food and some about vehicles. Because the nouns within a topic are interchangeable in the templates, the result is cleaner than real text would be. The method is the same one used on millions of real sentences. We also drop a few very common function words first, so that "the" does not swamp every row (see stop words).

Python Build a co-occurrence matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each row is a word's profile. cat co-occurs with ate, fed, grass and morning; car with light, stopped, down and drove. The matrix is symmetric because the window looks the same distance in both directions. Notice that train and car share light, stopped, down and drove, which is exactly what makes them look related. (Your counts match the output above because the random generator is seeded.)

The matrix is also wide and mostly zeros. With a real vocabulary of 100,000 words it would have 10 billion cells, so real systems store it as a sparse matrix or compress it, which is the subject of the next lessons.

Comparing rows with cosine

Each row is a vector, so you can compare two words with cosine similarity: the angle between their count vectors. Rows that put weight on the same context words point in the same direction, so the cosine is close to 1; rows with nothing in common have a cosine of 0. This is the same measure you used for documents. Only the rows now stand for words instead of documents.

The problem with raw counts

Raw counts have a flaw you already know from TF-IDF: frequent context words dominate. A word like ate or morning appears with many different words, so it contributes a lot to every dot product without telling us much. What we want to know is not "how often did c appear near w?" but "did c appear near w more often than chance?"

Pointwise mutual information (PMI) answers exactly that:

TEXT
PMI(w, c) = log2( P(w, c) / ( P(w) * P(c) ) )

P(w, c) is the share of all co-occurrences that are the pair (w, c), and P(w), P(c) are the shares of each word on its own. If the words are independent, the ratio is 1 and the PMI is 0. If they occur together twice as often as chance, the PMI is 1. If they occur together less than chance, the PMI is negative.

Negative values are unreliable: to learn that two words appear together less than expected you need a huge amount of data, and for a zero count the logarithm is minus infinity. The standard fix is positive PMI (PPMI): replace every negative value, and every minus infinity, with 0.

PMI has a known bias: it favors rare events. A pair seen once, whose words are both rare, gets a very high PMI. Real systems smooth the context counts (for example by raising them to the power 0.75) or discard very rare words. With a toy corpus whose counts are this regular the effect is mild, but remember it when you apply PPMI to real text.

Raw counts versus PPMI

The second example puts both representations side by side. It finds the nearest neighbors of a word by cosine similarity, prints a few raw counts next to their PPMI values, and finally varies the window size. It rebuilds the same corpus, so it is self-contained.

Python Raw counts, PPMI and window size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For cat, the nearest neighbors are horse, rabbit and dog, the other animals, with raw counts and with PPMI alike. For bread they are the other foods. Nobody told the program that these are animals or foods: it only saw that they appear in the same kinds of sentences. In this corpus the raw counts already work well, but PPMI tightens the picture: the similarity between cat and dog rises from 0.51 to 0.64, and horse rises from 0.85 to 0.91. The fourth neighbor in both lists is a context word rather than a noun, and its identity changes (fed with raw counts, ate with PPMI). That is a reminder that a word's nearest neighbors always include some words that are merely associated with it.

The middle block shows the weighting directly. lunch occurs 9 times near bread and gets a PPMI of 2.73: lunch appears only in food sentences, so the pair is far above chance. market appears near both foods and vehicles (the template "he bought the ___ at the market" is shared), so it occurs 4 times near bread and 3 times near car, with modest PPMI values of 1.45 and 1.67. The context word that is specific to a word gets the big score.

The last block shows the window size trade-off. With a window of 1, the other vehicles score between 0.90 and 0.98 against car; with a window of 4 they drop to between 0.78 and 0.94, and topical context words such as down and drove climb into the top five. In general, small windows capture syntactic similarity (words that can substitute for each other) and large windows capture topical similarity (words that appear in the same subject area). On a toy corpus the effect is small, but on real text it is large and well documented.

Limits of this approach

  • Size. The matrix grows with the square of the vocabulary. You need to compress it.
  • Sparsity. Most cells are zero, and rare words have rows with only a few non-zero entries, so their vectors are unreliable.
  • One vector per word. bank as a river bank and bank as a financial institution get a single blended row. This is the problem of word senses.
  • Association is not synonymy. Words that appear in the same contexts can be opposites: hot and cold appear in nearly identical sentences. "Similar" means "used alike", not "means the same".

The cure for the first two limits is to squeeze each row into a short, dense vector of, say, 100 numbers. That is what word embeddings do.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the distributional hypothesis say?

    The hypothesis says meaning can be read from the contexts a word appears in, so words used in similar contexts are similar in meaning.
  2. 2. In a word-context co-occurrence matrix with a window of 2, what does the cell for row "cat" and column "fed" count?

    Each cell counts the occurrences of the context word within the window around the target word, on either side.
  3. 3. Why is PPMI usually a better weighting than raw counts?

    PMI compares the observed co-occurrence with what independence would predict, so frequent but uninformative context words are down-weighted. The positive version also clips negative values to 0.
  4. 4. You build vectors with a window of 1 and again with a window of 10. What difference do you usually expect?

    Close neighbors reflect grammatical role and substitutability, while distant words reflect the general subject area.
  5. 5. Which problem do count-based word vectors NOT solve?

    Each word gets exactly one row, so all senses of an ambiguous word are blended together.

Summary

  • The distributional hypothesis: a word's meaning can be approximated by the contexts it appears in.
  • A co-occurrence matrix counts, for each word, which context words appear within a window around it.
  • Rows are vectors, and cosine similarity between rows measures how similar two words are.
  • Raw counts overweight frequent context words; PMI compares counts to chance, and PPMI clips negatives to 0.
  • Small windows favor syntactic similarity, large windows topical similarity.
  • Count matrices are huge, sparse and give one vector per word, which motivates dense embeddings.

Next, learn how to train short dense vectors in Word Embeddings and Word2Vec.