Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 38 ยท Projects and Practice

Project: Semantic Search with Embeddings

Build a semantic search engine: chunk documents, embed with mean pooling, rank by cosine similarity, evaluate recall@k, and scale up. Real code plus a runnable numpy stand-in.

Keyword search finds documents that contain your words. Semantic search finds documents that mean what you asked, even with different words: "how do I get my money back" should find the page titled "Refund policy". It works by turning text into vectors (embeddings) so that similar meanings land near each other, then ranking by distance.

This project builds the whole pipeline: chunk, embed, index, search, evaluate.

Embedding models need transformers or sentence-transformers and PyTorch, which do not run on this site. The real model code is read-only. The runnable examples use a hashed character n-gram vectorizer as a stand-in embedding: it captures spelling overlap, not true meaning, so it will not find "money back" from "refund". Everything else (pooling, normalization, cosine ranking, chunking, recall@k) is exactly what you would use with a real model.

From tokens to one vector

An encoder returns one vector per token. A search index needs one vector per text, so you pool. The common recipe, used by most sentence-embedding models, is mean pooling over the real tokens (ignoring padding using the attention mask), then L2-normalization so that a dot product equals cosine similarity.

Real code (read-only):

Python
import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

name = "sentence-transformers/all-MiniLM-L6-v2"
tok, model = AutoTokenizer.from_pretrained(name), AutoModel.from_pretrained(name).eval()

@torch.inference_mode()
def embed(texts: list[str]) -> torch.Tensor:
    batch = tok(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
    hidden = model(**batch).last_hidden_state                  # (batch, tokens, dim)
    mask = batch["attention_mask"].unsqueeze(-1).float()       # 1 for real tokens
    pooled = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    return F.normalize(pooled, dim=-1)

Some models want a prefix on queries and documents (for example query: and passage:), others use CLS pooling instead of mean pooling. Always follow the model card. The pooling step with a mask is the part people get wrong, so here it is in numpy.

Python Mean pooling with an attention mask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Row 0 changes when you forget the mask (its padding positions contribute), while row 1, which has no padding, is unchanged. That is why padding differences between batches would otherwise change your embeddings.

Chunking documents

Models have a maximum length, and one vector for a 20-page document blurs everything. Split documents into chunks of a few hundred tokens, with a little overlap so a sentence cut at a boundary still appears whole in one chunk. Store with each chunk its source and position so you can show results and link back.

A complete mini search engine

Python Stand-in search engine: chunk, embed, index, query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The queries share words with their target documents, so even the character stand-in finds them. A real embedding model also finds "money back" for "refund"; try that query on a real model, not on this stand-in.

Evaluating retrieval

Build a small query set: 50 to 200 real queries, each with the document or chunk that should be found. Then measure:

  • Recall@k: the fraction of queries whose correct item appears in the top k results. This is the headline number for a retriever.
  • MRR (mean reciprocal rank): the average of 1 divided by the rank of the first correct result.
  • nDCG when several results are relevant to different degrees.

Compare candidates on this set: a different embedding model, chunk size, overlap, or adding a keyword (BM25) component. Hybrid search, combining keyword and vector scores, often beats either alone because keywords catch exact names and codes that embeddings blur.

Scaling up

For thousands of chunks, M @ query on a numpy matrix is still fast. For millions, use an approximate nearest-neighbour index (FAISS, HNSW-based libraries or a vector database) that trades a little recall for large speed-ups. Other practical points:

  • Embed documents once and store the vectors; embed only the query at request time.
  • Keep the same model and settings for documents and queries. Changing the model means re-embedding everything.
  • Use float16 or int8 vectors to cut memory.
  • A re-ranker (a cross-encoder that scores query-document pairs) applied to the top 20 to 50 results improves precision at modest cost.
  • This is also the retrieval half of RAG: pass the top chunks to a generative model as context (generation and decoding).
flowchart LR
  D["Documents"] --> C["Chunk"]
  C --> E1["Embed chunks (once)"]
  E1 --> I["Vector index"]
  Q["User query"] --> E2["Embed query"]
  E2 --> S["Top-k by cosine"]
  I --> S
  S --> R["Optional re-ranker"]
  R --> O["Results or LLM context"]

Documents are embedded offline; only the query is embedded at request time.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use the attention mask during mean pooling?

    Padding tokens are not real text; including them would make the embedding depend on batch padding.
  2. 2. After L2-normalizing embeddings, what is the dot product of two vectors?

    For unit vectors the dot product equals the cosine of the angle between them.
  3. 3. What does recall@5 measure?

    It checks whether the right item is found among the first k results.
  4. 4. Why do chunks usually overlap a little?

    Overlap protects information that sits near chunk boundaries.
  5. 5. You switch to a different embedding model. What must you do?

    Vectors from different models live in different spaces and cannot be compared.

Summary

  • Embeddings come from pooling token vectors (mean pooling with the mask, or CLS) and normalizing; follow the model card for prefixes and pooling.
  • Chunk documents with a little overlap; index vectors once and embed only queries at request time.
  • Rank by cosine similarity, evaluate with recall@k and MRR on a real query set, and consider hybrid search and re-ranking.
  • Use approximate nearest-neighbour indexes when the collection grows large.

Next: Troubleshooting and practice