Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 Β· Cold Start and Production

ANN Search and Serving

Serve recommendations at scale: exact top-k with argpartition, approximate nearest neighbor ideas (IVF, HNSW, quantization), recall versus speed, caching and a retrieve-then-rerank flow.

A trained model is only useful if it can answer requests fast. A user opens the home page and your system has roughly 100 milliseconds in total, including network, feature lookups, ranking and rendering. If the catalog has 10 million items, scoring every item for every request is far too slow.

Embedding-based models solve this by turning recommendation into a nearest-neighbor search: find the item vectors with the highest dot product against the user vector. This lesson shows how exact search works, why it stops scaling, and how approximate nearest neighbor (ANN) indexes trade a little accuracy for a huge speedup. Then it connects search to the rest of the serving pipeline.

Exact top-k

Brute force is the baseline: multiply the user vector by the item matrix, then keep the best k. Sorting everything costs O(n log n); selecting the top k with a partial selection costs about O(n). NumPy offers argpartition for that.

Python Brute-force top-k with argpartition

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One request costs 100,000 multiplications of 32 numbers. At 10 million items and 128 dimensions that is a billion multiply-adds per user, and you have thousands of requests per second. Brute force on a GPU, with batching, still works for moderate catalogs, but at larger scale you need an index.

Approximate nearest neighbors

ANN indexes avoid looking at most items. They accept missing a few of the true top-k in exchange for speed. The usual measure is recall of the approximate top-k against the exact top-k at a given amount of work. The main families:

  • Partition-based (IVF, inverted file): cluster item vectors with k-means. At query time, find the few clusters closest to the query ("probes") and scan only the items in them. Increasing the number of probes raises recall and cost.
  • Graph-based (HNSW): connect each vector to near neighbors in a multi-layer graph; a query walks greedily through the graph toward closer points. Very fast with high recall, but uses more memory.
  • Hashing (LSH): hash vectors so that similar ones collide, then compare only within buckets.
  • Quantization (product quantization, PQ): compress vectors into short codes so that billions of them fit in memory and distances are computed from lookup tables. Often combined with IVF ("IVF-PQ").
  • Libraries and services: FAISS, ScaNN, Annoy, hnswlib, and vector databases such as Qdrant, Milvus, pgvector or Elasticsearch/OpenSearch with vector fields.

Inside an IVF index

Build Cluster the item vectors
Probe Pick the nearest few clusters
Scan Score only those items
Tune More probes, more recall

The knob is always recall versus work.

1 / 2

Build your own IVF index

The code clusters 20,000 item vectors into 64 groups with a few k-means rounds, then answers queries by scanning only the p nearest clusters. For each setting it reports the average recall@10 compared with brute force and the fraction of items actually scored.

Python IVF: recall versus work

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from top to bottom: with one probe you scan about 2% of the catalog and still find about 91% of the true top 10; with four probes recall is about 0.99 while only about 6% of the items are scored. This data is strongly clustered, so IVF works unusually well; on less clustered embeddings you need more probes. That trade-off, set per application, is the heart of ANN tuning. Real systems also note that dot product is not a metric: items with large norms can dominate. Libraries handle this by normalizing vectors (cosine) or by a transformation that turns maximum inner product search into nearest-neighbor search.

The serving pipeline

ANN is one block in a larger flow, the candidate generation and ranking pipeline:

  1. Request arrives. Fetch user features and recent history (from a feature store or cache).
  2. Compute the user embedding with the user tower (milliseconds).
  3. Retrieve candidates: ANN search returns the top few hundred items. Often several retrievers run in parallel (embedding search, popular in region, recently viewed, trending) and their results are merged.
  4. Filter: remove items the user has seen, bought or blocked, out-of-stock items and policy violations. Over-fetch candidates so that filtering still leaves enough.
  5. Rank: a heavier model scores the few hundred with rich features (context, cross features between user and item).
  6. Re-rank: apply business rules, diversity (for example MMR) and exploration slots.
  7. Log: store what was shown (the impressions), the position and the scores. These logs are the future training data.
flowchart LR
  R["request"] --> UE["user embedding"]
  UE --> ANN["ANN retrieval<br/>~500 candidates"]
  P["popular / recent<br/>retrievers"] --> M["merge + filter"]
  ANN --> M
  M --> RK["ranking model"]
  RK --> RR["re-rank + rules"]
  RR --> OUT["top 10 shown"]
  OUT --> LOG["impression log"]

A typical online path. Each stage narrows the list.

Making it fast and cheap

  • Precompute what you can. Item embeddings and indexes are rebuilt offline (hourly or daily). User recommendations for stable segments can be precomputed in batch and cached.
  • Cache. Cache per-user results for a short time, and cache popular lists for everyone. Remember that personalization and caching pull in opposite directions.
  • Approximate then exact. ANN finds candidates; the ranker re-scores them exactly.
  • Freshness. New items must reach the index quickly (streaming updates or a small "fresh" index merged at query time), or cold-start items stay invisible.
  • Monitoring. Track latency percentiles (p50, p99), candidate counts after filtering, empty results, index age and the coverage of recommended items.
  • Fallbacks. If a service times out, return popular items rather than an error.

Check your understanding

Quick check Β· 4 questions

of 4 answered You scored / 4 Β· perfect!

  1. 1. Why does brute-force search stop scaling for large catalogs?

    Every request multiplies the user vector with all item vectors, which becomes too costly at millions of items.
  2. 2. In an IVF index, what does increasing the number of probes do?

    More probed clusters mean more candidates are scored, so recall rises along with work.
  3. 3. How is the quality of an ANN index usually measured?

    ANN is an approximation of exact search, so overlap with the exact result at a given cost is the key metric.
  4. 4. Why fetch more candidates than you finally show?

    After filtering and re-ranking only a few remain, so retrieval has to over-fetch.

Summary

  • Embedding models turn recommendation into maximum inner product search over item vectors.
  • Exact top-k uses partial selection but scales linearly; ANN indexes (IVF, HNSW, LSH, PQ) trade a little recall for big speedups.
  • The tuning knob is recall versus work, measured against exact search.
  • Serving is a pipeline: embed, retrieve, merge and filter, rank, re-rank, log.
  • Precompute, cache, keep indexes fresh, monitor latency and always have a fallback.

Next: Feedback Loops, Bias and Fairness