Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 36 ยท Embeddings and Similarity

Embedding Geometry and the Curse of Dimensionality

See why distances behave strangely in high dimensions, what hubness and anisotropy do to nearest-neighbor search, and how centering and thresholds keep scores meaningful.

Our intuition about distance comes from two and three dimensions. Embeddings live in hundreds or thousands. In that world several odd things happen: all points become almost equally far apart, a few "hub" vectors show up as neighbors of everything, and cosine scores for unrelated texts are not near zero. None of this breaks vector search, but it explains puzzling behavior (why every score is between 0.6 and 0.8, why one chunk keeps appearing in the results) and it explains why indexing needs clever algorithms.

You will test each effect with NumPy.

Distances concentrate

Draw random points in a unit cube and compare the nearest and farthest neighbor of a query. In 2 dimensions the nearest is much closer than the farthest. As dimensions grow, the gap shrinks: everything is roughly the same distance away. This is called distance concentration.

Python Nearest versus farthest, by dimension

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ratio falls toward 1 as the dimension grows. For random data the "nearest" neighbor is barely nearer than a typical point, so you cannot tell neighbors apart from noise.

Real embeddings are not uniformly random: they sit on a much lower-dimensional surface (texts have limited topics), so nearest neighbors remain meaningful. Researchers call that the intrinsic dimension, and it is usually far below the nominal number. That is why ANN indexes work on real data (next chapter) and why a random-vector benchmark can be misleadingly hard.

Random vectors are nearly orthogonal

In high dimensions two random directions are almost always close to a right angle. Their cosine concentrates around 0 with a spread of about 1 / sqrt(dims):

Python Cosine of random vectors shrinks toward zero

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The consequence is useful: if two embeddings have a cosine of, say, 0.5 in 1536 dimensions, that is many standard deviations away from what chance gives, so it is a strong signal. A cosine of 0.5 would mean almost nothing in three dimensions.

Anisotropy: unrelated texts are not at cosine 0

Real text models usually do not spread vectors evenly in all directions. Their outputs share a common offset (a mean vector) and occupy a narrow cone, a property called anisotropy. The visible effect: the cosine between unrelated sentences is often 0.6 or 0.7, not 0, and relevant pairs land between 0.8 and 0.9. Scores are compressed into a narrow band.

Two practical lessons follow. First, raw scores are not probabilities: a threshold like 0.75 only has meaning for one model on one dataset (thresholds). Second, one cheap fix is centering: subtract the mean vector of your corpus from every embedding, then re-normalise. This removes the shared offset and spreads the scores.

Python Stand-in: centering removes a shared offset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before centering, different topics already look similar (the shared offset dominates); after centering, different topics fall near or below zero while same-topic pairs stay high. Centering is cheap, but it must be applied the same way to queries (subtract the corpus mean), and it changes your thresholds, so re-tune after switching it on. Test it on your evaluation set before adopting it: some models are already well-spread and gain little.

Hubness: some vectors are neighbors of everyone

In high dimensions a few points become hubs: they appear in the top-k lists of many queries, while others (anti-hubs) appear in almost none. You may see it in practice as "this one chunk shows up for every question". Hubs tend to be vectors near the center of the data.

Python Counting how often each point is a neighbor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In 3 dimensions the most popular point is listed 20 times and every point is listed at least once. At 500 dimensions one hub is listed 201 times (the points near the center of the cloud win) while 26% of points are never listed at all. For RAG the remedies are practical: re-rank the candidates, use hybrid search, avoid chunks that are too generic (they behave like hubs), and add metadata filters.

What it means in practice

flowchart TD
    A["High-dimensional embeddings"] --> B["Distances concentrate"]
    A --> C["Anisotropy: narrow score band"]
    A --> D["Hubness: popular chunks"]
    B --> E["Exact search stays costly, ANN indexes needed"]
    C --> F["Calibrate thresholds, consider centering"]
    D --> G["Rerank, filter, dedupe generic chunks"]

Geometry effects and the habits they motivate.

  • Do not eyeball raw scores. Compare scores within one query's results, and calibrate thresholds on your own data.
  • Use relative decisions when you can: top-k, or "score within 0.05 of the best", rather than a fixed cutoff.
  • Test centering and normalisation on your evaluation set.
  • Inspect popular chunks. If one chunk appears for many unrelated queries, fix or remove it.
  • Do not trust random benchmarks. An index tuned on random vectors can behave very differently on real embeddings.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What happens to the ratio of farthest to nearest distance for random points as dimensions grow?

    This is distance concentration: in high dimensions the contrast between near and far shrinks.
  2. 2. Two random unit vectors in 3000 dimensions have a cosine close to:

    Random directions are almost orthogonal in high dimensions; cosine has mean 0 and spread about 1 over the square root of dimensions.
  3. 3. Why is a fixed similarity threshold like 0.75 not portable between models?

    Score bands differ per model and dataset, so a threshold has to be calibrated on your own data.
  4. 4. What is a 'hub' in a vector index?

    Hubness means some points are nearest neighbors of a disproportionate number of queries.
  5. 5. What does centering do to embeddings?

    Removing the common mean spreads out the vectors so unrelated items score lower.

Summary

  • In high dimensions distances concentrate and random vectors are nearly orthogonal, so a decent cosine is a real signal.
  • Real embeddings are anisotropic: unrelated texts score well above 0, so calibrate thresholds per model.
  • Centering by the corpus mean can spread scores; test it before you adopt it.
  • Hubness makes a few chunks appear for many queries; reranking, hybrid search and filters help.
  • Real data has a low intrinsic dimension, which is what makes approximate indexes work.

Next: Exact Search, ANN and the Recall Tradeoff