Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 36 ยท Embeddings and Similarity

What Are Embeddings?

Understand embeddings as points in space where similar things sit close together, then build word vectors from scratch with co-occurrence counts and SVD in NumPy.

An embedding is a list of numbers that represents something (a word, a sentence, a document, an image) in such a way that similar things get similar numbers. You can picture it as a map: every item is a point, and items with related meaning are close together. Vector search is then just "find the points near this point".

Nothing about this is magic. The hard part is producing the numbers so that distance matches meaning. Modern models learn this from huge amounts of text. In this lesson you will see the idea with features you choose by hand, and then build real, learned word vectors yourself from counting words.

Meaning as coordinates

Imagine describing animals with three numbers: how big, how fast, how domesticated (each from 0 to 1).

Python Hand-made embeddings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cat's nearest neighbour is the dog, the elephant is far from everything (its nearest neighbour is the horse, at distance 0.87), and the cheetah, fast and wild, lands closest to the mouse and then the horse. Nobody wrote a rule saying "cats are like dogs". It falls out of the geometry. A learned embedding does the same thing with hundreds of dimensions that no person named.

What an embedding is

A point Each item is a position in space.
Close = similar Related items sit near each other.
Just numbers Typically 100 to 4,000 of them.
1 / 2

Words are known by the company they keep

How can a computer learn that "cat" and "dog" are related with nobody labelling anything? Linguists have a famous answer: a word's meaning is reflected by the words that appear around it. Cats and dogs both "sleep", "chase" and "drink milk or water". Cars and trucks both appear near "highway" and "garage".

So we can build embeddings with three steps:

  1. Count, for each word, which words appear within a few positions of it (a co-occurrence matrix).
  2. Turn raw counts into a score of how surprisingly often two words meet (PPMI, positive pointwise mutual information).
  3. Squash each long row into a short vector with SVD, a matrix factorisation that keeps the strongest patterns.

That is a classic recipe for word vectors, and it is the intuition behind the neural methods that replaced it.

flowchart LR
    T["Text"] --> C["Co-occurrence counts"]
    C --> P["PPMI scores"]
    P --> S["SVD (keep k dimensions)"]
    S --> V["Word vectors"]

Statistical word embeddings in three steps.

Building word vectors from 23 sentences

The corpus below is tiny and artificial, with three themes: pets, vehicles and cooking. Real models train on billions of words, but even here you will see structure appear.

Python Word vectors from co-occurrence and SVD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The neighbours of car include train, station, bus and highway, all vehicle words; bread sits near sold, chef, cooked and baked; dog is close to cat, garden and barks. The program was never told these categories. It found them only from which words share contexts.

Real embedding models differ in method (neural networks trained to predict words or to separate matching from non-matching text pairs) and in scale, but the principle is the same: shared context becomes shared position.

Properties of a good embedding space

  • Locality: similar items are near each other.
  • Fixed size: every item, short or long, becomes the same number of dimensions, so a database can compare them.
  • Dense: most values are non-zero (unlike one-hot or TF-IDF vectors, which are mostly zeros).
  • Task-shaped: a model trained to find answers to questions places a question near its answer; a model trained for duplicate detection places paraphrases near each other. Match the model to your task.

From words to sentences and documents

A sentence embedding is not simply "the words' vectors added up", though averaging word vectors is a surprisingly decent baseline. Modern text embedding models read the whole text with a transformer network and output one vector for the full passage, taking word order and context into account ("bank" of a river versus "bank" account).

Python A baseline: average the word vectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query "dog sleeps" shares no words with "puppy sleeps" except "sleeps", yet the pet sentences clearly outrank the vehicle ones because dog and puppy have similar vectors.

Where embeddings are used in RAG

In a RAG system you embed every chunk once at ingestion and every question at query time with the same model. The nearest chunks to the question become the context for the LLM. The quality of that map decides how often the right chunk is found, so choosing and testing the embedding model matters more than almost any other decision (embedding models).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What property should an embedding have?

    Locality is the defining property: distance in the space reflects similarity in meaning.
  2. 2. In the co-occurrence approach, why do 'cat' and 'dog' end up with similar vectors?

    Shared contexts produce similar rows in the co-occurrence matrix and therefore similar vectors.
  3. 3. What does the SVD step do in the pipeline?

    SVD factorises the PPMI matrix; keeping the top k components gives dense, low-dimensional word vectors.
  4. 4. In RAG, which model must embed the user's question?

    Query and chunk vectors must come from the same space to be comparable.

Summary

  • An embedding places an item as a point so that distance reflects similarity.
  • Words with similar contexts get similar vectors; counting, PPMI and SVD already reveal this.
  • Neural models do the same at scale and embed whole passages.
  • Embeddings are dense, fixed-size and model-specific: always use one model for documents and queries.

Next: Similarity Metrics