Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 63 ยท Word Meaning and Embeddings

Sentence and Document Embeddings

Turn sentences and documents into single vectors. Average word vectors, weight them with IDF, see what averaging misses, and build a tiny semantic search.

Word vectors describe words, but most NLP tasks work on larger units: a tweet, a review, a support ticket, a web page. To classify, cluster or search them, you need one vector per text. A TF-IDF vector is one answer, but it is long, sparse and blind to meaning: "car" and "automobile" never match. A sentence embedding (or document embedding) is a short dense vector that places texts with similar meaning close together, even when they share no words.

This lesson starts with the simplest recipe, averaging word vectors, and examines exactly what it keeps and loses. Then it builds a miniature semantic search and compares it with keyword search. The vectors in the examples come from the small SVD embeddings of the previous lesson, standing in for the pre-trained vectors you would use in a real project.

Averaging word vectors

The baseline is almost embarrassingly simple: look up the vector of each word in the text and take the mean. If dog and field point toward "animals and the countryside", their average also points that way. The sentence vector lives in the same space as the word vectors, so cosine similarity compares sentences just as it compares words.

Averaging, step by step

Tokenize Split the text into words.
Look up Fetch each word's vector.
Average Add them up, divide by the count.
Compare Use cosine between texts.
1 / 2

The plain mean has a weakness. A sentence of ten words where six are filler (the, of, is) gives the fillers most of the vote. Two remedies are standard. The first is to remove stop words. The second is to weight each word by its inverse document frequency, as in TF-IDF, so that common words count for little and distinctive words count for a lot. A well-known variant called SIF (smooth inverse frequency) uses weights of the form a / (a + p(word)) and also subtracts the first principal component, and it is a surprisingly strong baseline.

A first sentence embedder

The example rebuilds the toy word vectors (five dimensions, PPMI plus SVD) and defines embed(), which averages the vectors of the words it knows. Words missing from the vocabulary are skipped, which turns out to matter. The script compares three sentence pairs with a plain mean and an IDF-weighted mean, then probes what averaging cannot see.

Python Average word vectors into a sentence vector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first pair has no word in common (apart from the, which has no vector), yet the sentences get a cosine of 0.98: both are about animals on a farm. The second pair, an animal sentence against a vehicle sentence, scores 0.01. That is the promise of embeddings: similarity of meaning, not of spelling.

The third pair shows weighting at work. The sentences share the word morning, but one is about a horse and the other about bread. The plain mean gives 0.67, partly because of that shared word; the IDF-weighted mean lowers it to 0.62, since morning appears in many sentences and counts for less. The effect is small here because this toy vocabulary is nearly uniform. In real text, where frequent words are far more frequent, the gap is large.

The last block lists the blind spots. Averaging is a bag of vectors, so it ignores order: "the cat fed the dog" and "the dog fed the cat" get cosine 1.00, although the meaning is reversed. Negation also vanishes: not has no vector in this vocabulary, so it is skipped and "the soup is warm" and "the soup is not warm" look identical. Even a model that did have a vector for not would only shift the average slightly, not flip the meaning.

The most popular use of sentence embeddings is semantic search. Embed every document once and store the vectors. When a query arrives, embed it with the same function and return the documents with the highest cosine similarity. No shared words required.

flowchart LR
    D["Documents"] --> E1["Embed each one"]
    E1 --> I["Vector index"]
    Q["Query"] --> E2["Embed the query"]
    E2 --> N["Nearest vectors"]
    I --> N
    N --> R["Ranked results"]

Documents are embedded ahead of time; only the query is embedded at search time.

The next example compares this with TF-IDF keyword search over six short documents, one of which is about Kubernetes, a word our toy vocabulary has never seen. It also tries a simple hybrid that averages the two scores, which is what many production systems do.

Python Keyword search versus embedding search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each query shows a different failure mode:

  • "the horse slept": keyword search finds the rabbit sentence because it contains slept (0.53). It also gives a second place to the mechanic sentence for the word the alone, which is noise. Embedding search ranks the cat sentence first (1.00) and the rabbit sentence second (0.97): it knows that horse is related to cat and rabbit, though it cannot tell that the rabbit sentence also shares slept.
  • "a bowl of rice": no document contains bowl or rice, so every keyword score is 0.00. Embedding search returns "we shared the soup" at 0.98, since rice and soup appear in the same kinds of sentences.
  • "kubernetes cluster": the keyword search finds the right document at once (0.79). The embedding has no vector for either word and returns all zeros, so the ranking is meaningless. Words that are rare, new, or in a technical vocabulary are exactly where embeddings built from a fixed vocabulary break down.

The hybrid score gets the right document first in all three cases, which is why combining keyword and embedding search is so popular. The weights (here 50/50) are tuned on your own data.

Beyond averaging

Averaging works, but it has a ceiling. The better options are:

  • Doc2Vec (paragraph vectors): extends Word2Vec by learning an extra vector per document during training. Useful, but the document vectors must be trained for the exact corpus, and new texts need an inference step.
  • Sentence encoders: neural networks trained directly to produce one vector per sentence so that paraphrases land close together and unrelated sentences land far apart. Sentence-BERT is the best-known family. They see word order and context, so "the dog bit the man" and "the man bit the dog" get different vectors.
  • Embeddings from LLM providers: hosted APIs that return a vector (commonly 384 to 3,072 dimensions) for any text you send.

Real usage with the sentence-transformers library looks like this (read-only: the site cannot download model weights):

Python
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")   # about 90 MB, 384 dimensions
docs = ["A cat ran across the field", "He parked the truck", "Deploy the Kubernetes cluster"]
doc_vectors = model.encode(docs, normalize_embeddings=True)

query_vector = model.encode("the horse galloped", normalize_embeddings=True)
scores = util.cos_sim(query_vector, doc_vectors)[0]
best = scores.argmax().item()
print(docs[best], float(scores[best]))

Because the vectors are normalized to length 1, cosine similarity is just a dot product, so a whole collection can be searched with one matrix multiplication. For millions of documents, specialized vector indexes (approximate nearest-neighbor search, in libraries such as FAISS or in databases with vector support) avoid comparing the query with every document. We come back to this in Retrieval and RAG.

Choosing a representation

  • TF-IDF: fast, exact-word matching, great for keyword-heavy tasks, a strong baseline for classification with enough labeled data.
  • Averaged word vectors: cheap meaning-aware baseline; blind to order and negation.
  • Sentence encoders: best quality for similarity and search; they need a model download and more compute, and still have domain blind spots.
  • Hybrid: combine keyword and embedding scores when both exact terms and meaning matter.

Always evaluate on your own queries. A model that wins on a benchmark can lose on your domain's jargon.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How does the simplest sentence embedding turn a sentence into one vector?

    Averaging keeps the vector size fixed no matter how long the sentence is, and puts the sentence in the same space as the words.
  2. 2. Why does averaging word vectors give the same result for "the cat fed the dog" and "the dog fed the cat"?

    A mean is order-free. It sees the same bag of vectors in both sentences, so it cannot represent who did what to whom.
  3. 3. What does IDF weighting do when averaging word vectors?

    Each vector is multiplied by the word's inverse document frequency before averaging, so common words contribute little.
  4. 4. A user searches for "kubernetes cluster", but the embedding model's vocabulary has never seen either word. What happens?

    Words missing from the vocabulary contribute nothing to the average. Keyword matching needs no vocabulary of vectors, which is a reason to use hybrid search.
  5. 5. What is the main advantage of semantic search over keyword search?

    Embeddings place texts with related meaning close together, so "bowl of rice" can match a document about soup. It is not always faster, and it can still return irrelevant results.

Summary

  • A sentence or document embedding is one dense vector per text, so texts can be compared, clustered and searched by meaning.
  • Averaging word vectors is a strong cheap baseline; weighting by IDF and dropping stop words improve it.
  • Averaging ignores word order and negation, and silently skips unknown words.
  • Semantic search embeds documents once and compares the query's vector with them by cosine similarity; it finds matches without shared words.
  • Keyword search is better for rare, exact terms; hybrid scores combine both strengths.
  • Sentence encoders such as Sentence-BERT see context and order and are the usual choice in production.

Next, move from fixed vectors to learned networks in Neural Networks for Text.