Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 53 ยท Tokens and Embeddings

Embeddings: Meaning as Numbers

Understand embeddings: how words and sentences become vectors, why similar meanings land close together, and build a tiny embedding with SVD in NumPy.

A token ID is just a label. Token 5,021 is not "closer" to token 5,022 in any useful way, so a model cannot do arithmetic with IDs and learn anything about meaning. An embedding fixes this: it replaces each token (or a whole sentence, or an image) with a list of numbers called a vector. The vectors are arranged so that things with similar meaning get similar numbers.

Embeddings are the bridge between text and mathematics. Inside a language model, every token is embedded before the transformer sees it. Outside a model, embedding models turn documents into vectors for search, clustering and retrieval-augmented generation. If you know NumPy arrays from NumPy Basics, you have all the math you need.

Meaning as a position in space

Imagine describing animals with three hand-picked numbers: how big, how dangerous, how domestic. Each animal becomes a point in 3D space, and animals that are alike sit near each other.

Python Hand-made embeddings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Cat and dog are close, lion and tiger are close, and a cat is far from a lion. Nobody told the computer what a "cat" is. We only gave it coordinates, and distance now means dissimilarity.

Real embeddings work the same way, with two differences. Nobody hand-picks the features: a neural network learns them from data. And there are far more of them. The number of coordinates is the dimension of the embedding: small word models use 50 to 300, and modern text embedding models commonly use several hundred to a few thousand. Individual dimensions rarely have a clean human meaning, such as "danger". The meaning is in the overall pattern.

What an embedding is

A vector A fixed-length list of numbers.
A position Each item is a point in space.
Closeness Similar meaning means nearby points.
1 / 2

Learning meaning from context

How can a computer learn that "cat" and "dog" are alike without a dictionary? From the company they keep. Both appear near words like "the", "chases", "drinks" and "pet". A very old and surprisingly powerful idea is to count which words appear near which, then compress those counts.

The recipe:

  1. Build a co-occurrence matrix: row = a word, column = another word, cell = how often they appear within two positions of each other.
  2. Compress the matrix with SVD (singular value decomposition), keeping only the strongest few directions. Each word's row becomes a short vector.

Neural methods such as word2vec, and the embedding tables inside transformers, reach a similar result by gradient descent instead of SVD. The next example uses a tiny corpus of 18 sentences, so results are rough. Real corpora have billions of words.

Python A tiny embedding from co-occurrence counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The nearest neighbors of "cat" are other animals, and "king" lands next to "queen". The corpus never defined any of these categories. The model only saw which words appear in similar sentences, and that was enough. (Words with the same role, like "king" and "boy", also land fairly close, so the vectors capture grammar-like roles as well as topics.)

The similarity measure here is the dot product of unit-length vectors, called cosine similarity. The next lesson covers it properly.

Vector arithmetic and analogies

A famous result from word embeddings is that directions in the space can carry meaning. The offset from "man" to "woman" is similar to the offset from "king" to "queen". So king - man + woman should land near queen.

Python Analogy: king - man + woman

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treat the results as a demonstration, not a guarantee. We exclude the three input words from the answer (otherwise "king" often wins trivially), and with such a small corpus the ranking is fragile. With millions of sentences, analogies work more reliably, but they are still a curiosity rather than a feature to depend on.

Embeddings of sentences and documents

Word embeddings give one vector per word, so "bank" has the same vector in "river bank" and "bank account". Modern models fix this with contextual embeddings: the vector for a token depends on the whole sentence. An embedding model goes one step further and pools these into a single vector for a whole sentence or paragraph, trained so that texts with similar meaning get nearby vectors, even when they share no words.

Python
# READ-ONLY: the shape of a typical embedding API call (no network on this site)
response = client.embeddings.create(
    model="some-embedding-model",
    input=["How do I reset my password?", "I forgot my login credentials"],
)
vectors = [item.embedding for item in response.data]   # two lists of floats
print(len(vectors[0]))                                  # the model's dimension

Two things to remember when using embedding models:

  • Use the same model for everything you compare. Vectors from different models live in different spaces and cannot be compared.
  • Dimension is a trade-off. More dimensions can capture more nuance but cost more memory and search time. Some models let you truncate their vectors to a smaller size.

Embeddings are also learned for images, audio and code, and some models place text and images in one shared space. That is how a text query can find a matching picture. See Audio, Video and Multimodal Models.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are token IDs alone a poor input for learning meaning?

    An ID is an arbitrary label. Embeddings replace it with a vector where distance reflects similarity of meaning.
  2. 2. What does the dimension of an embedding mean?

    A 768-dimensional embedding is a vector of 768 numbers per item.
  3. 3. In the co-occurrence demo, why do 'cat' and 'dog' get similar vectors?

    Words used in similar contexts have similar co-occurrence rows, so after compression their vectors are close.
  4. 4. Can you compare a vector from model A with a vector from model B?

    Even with equal dimensions, the coordinates mean different things in different models.

Summary

  • An embedding maps a token, sentence, image or document to a vector of numbers.
  • Similar meaning means nearby vectors, so distance and angle become tools for reasoning about meaning.
  • Embeddings are learned from context: words used in similar situations end up close together.
  • A co-occurrence matrix plus SVD gives a small working embedding; neural networks learn better ones at scale.
  • Directions in the space can carry relations, which is why analogies like king - man + woman can work, but only approximately.
  • Always compare vectors that come from the same embedding model.

Next: measure closeness and search with it in Cosine Similarity and Semantic Search.