Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 36 ยท Embeddings and Similarity

Embedding Models and Dimensions

Learn how text embedding models differ, what dimensionality costs in memory and speed, how truncation works, and how to choose and test a model for your RAG data.

Every vector database project starts with one decision that is painful to change later: which embedding model do you use? Once millions of chunks are embedded, switching models means re-embedding everything, because vectors from different models live in different spaces and cannot be compared.

This lesson explains what differs between models, why the number of dimensions drives your storage and latency bill, how you can shrink vectors safely, and how to pick a model with a test instead of a leaderboard.

A note on what runs here: the real hosted and open-weight models cannot run inside this page, so the examples use small stand-ins (hashed word vectors, SVD vectors) that behave the same way where it matters: dimensions, memory, truncation and testing. Each stand-in is labelled.

What differs between models

Property Why it matters
Dimensions Vector length, commonly 384, 768, 1024, 1536 or 3072. More dimensions can hold more nuance but cost more memory and time.
Max input tokens Text beyond the limit is cut off or rejected. It caps your chunk size (chunking).
Languages Some models are English-only, others multilingual (a French question can match an English passage).
Training task Retrieval models place questions near their answers; similarity models place paraphrases together.
Query vs document modes Some models expect a prefix such as "query:" or "passage:", or separate input types. Skipping it quietly lowers quality.
Normalisation Whether output is unit length decides which metric is right.
Cost and hosting A paid API (per token), or an open model you run yourself (per GPU hour, but data stays private).
Licence Open weights vary: some are fine for commercial use, some are not.

There are also two shapes of model worth naming. Dense models output the fixed-length vectors this track uses. Sparse models output a long mostly-zero vector over words or sub-words (like BM25 but learned), which pairs well with dense ones in hybrid search.

The memory arithmetic

A stored vector costs dimensions x bytes per number. A 32-bit float is 4 bytes. For a million chunks:

Python How many gigabytes is your index?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ten million chunks at 1536 dimensions in float32 need about 86 GB including overhead, far more than the same data at 384 dimensions (about 21 GB). Indexes that are searched fast usually sit in RAM, so dimensions translate directly into server size. Compression can shrink these numbers a lot, at a small quality cost.

More dimensions: is it better?

Usually a bit better, with diminishing returns. The extra dimensions give the model more room to separate similar topics, but beyond a few hundred dimensions many tasks gain little, while every dimension is paid for on every distance computation.

We can imitate the effect with a stand-in embedder: the hashing trick maps each word to a few random positions in a vector of chosen size. With few dimensions many words collide and meaning blurs; with more dimensions collisions fade.

Python Stand-in: collisions shrink as dimensions grow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This stand-in only knows shared words, so it is far weaker than a real model, but you can see the trend: with 4 dimensions it picks the wrong top result for 2 of 6 queries, and from 64 dimensions on it matches exact search every time. Real models show the same curve at larger scale.

Shrinking vectors: truncation

Some models are trained so that the first k dimensions already form a good smaller embedding (this training idea is often called Matryoshka representation learning, after the nesting dolls). You can then cut the vector to 256 dimensions, re-normalise, and keep most of the quality at a fraction of the cost. You must truncate only models that are documented to support it; for other models, the first coordinates are arbitrary and truncation hurts badly.

SVD-based embeddings have the same property naturally, because SVD sorts directions from strongest to weakest. Here is a stand-in with topic structure, truncated step by step.

Python Stand-in: truncating an ordered embedding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The curve climbs quickly (one dimension finds the right topic 32% of the time, four find it every time) and then stays flat: later dimensions mostly carry noise. That is the intuition behind truncating a good model to a smaller size.

Asymmetric search and prefixes

A question ("how do I reset my password?") and the passage that answers it ("To reset your password, open Settings...") look very different as text. Retrieval models are trained on such pairs and often treat the two sides differently:

  • Some use two modes on one model: input_type=query versus input_type=document in an API, or text prefixes like query: ... and passage: ... for open models.
  • Embedding a query as if it were a document usually still works, but ranks worse.

Always read the model card for the exact convention and apply it at both ingestion and query time.

Picking a model: test, do not guess

Public benchmarks (such as the MTEB leaderboard) are a useful shortlist, but they average over tasks that are not yours. A reliable process:

  1. Shortlist three or four models that fit your budget, languages and hosting rules.
  2. Build a small evaluation set: 50 to 200 real questions, each with the chunk that answers it.
  3. Embed your chunks with each model, run the questions, and compare retrieval metrics such as recall at 5.
  4. Weigh the difference against price, latency and dimensions. A 2-point recall gain rarely justifies four times the storage; a 15-point gain does.
  5. Record the model name and version with every vector, so you can migrate later.
flowchart LR
    S["Shortlist 3-4 models"] --> E["Embed your chunks"]
    E --> R["Run your questions"]
    R --> M["Compare recall@k, cost, latency"]
    M --> D["Choose and record version"]

Choosing a model with your own data.

Practical rules

  • One model per index. Never mix vectors from different models (or different versions) in one index.
  • Same model for queries. Documents and queries must come from the same space.
  • Mind the token limit. Chunk text to fit, or the end of each chunk is silently dropped.
  • Batch your calls. Embedding thousands of chunks one by one is slow; send batches.
  • Plan for change. Keep the raw text so re-embedding is possible, and store the model version.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can you not mix vectors from two different embedding models in one index?

    Each model defines its own coordinate system. Comparing vectors across models gives meaningless scores.
  2. 2. 10 million vectors of 1536 float32 dimensions versus 384 dimensions: roughly how do raw sizes compare?

    Size scales linearly with dimensions: 1536 / 384 = 4.
  3. 3. When is it safe to truncate an embedding to its first 256 dimensions?

    Truncation works when the model concentrates information in the leading dimensions, which has to be trained in.
  4. 4. What is the most reliable way to choose between two embedding models?

    Benchmarks are averages. Your own evaluation set measures what matters for your content.
  5. 5. What goes wrong if a model expects a 'query:' prefix and you leave it out?

    The model still returns vectors, but the query and document sides are no longer aligned as trained.

Summary

  • Models differ in dimensions, token limits, languages, training task, query/document modes and cost.
  • Memory grows linearly with dimensions and with bytes per number; plan the index size before choosing.
  • More dimensions help with diminishing returns; truncation is only safe for models built for it.
  • Follow the model's query and document conventions and never mix models in one index.
  • Choose by testing on your own question set, and record the model version with each vector.

Next: TF-IDF and LSA: Embeddings Without a Neural Network