Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 63 ยท Counting and Classic Representations

Document Similarity and Cosine Distance

Measure how similar two texts are. Compare cosine similarity, Jaccard and Euclidean distance on word vectors, and find the nearest documents with TF-IDF.

Once documents are vectors, a natural question follows: which two are most alike? Similarity is the engine behind "related articles", duplicate detection, recommendations, clustering and search. A search query is just a very short document, and the best results are the documents most similar to it.

The question is how to measure "alike". Different measures give very different answers on the same vectors, and the wrong choice can quietly ruin a system. This lesson compares the three you will meet most often: cosine similarity, Jaccard similarity and Euclidean distance, and ends with a nearest-document lookup using TF-IDF vectors.

Three ways to compare

Three yardsticks

Cosine The angle between two vectors.
Jaccard Shared words over all words.
Euclidean The straight-line distance.

Cosine ignores length, Euclidean does not.

1 / 2

Cosine similarity treats each document as an arrow from the origin and measures the angle between two arrows. It is the dot product of the vectors divided by the product of their lengths:

TEXT
cosine(u, v) = (u ยท v) / (|u| * |v|)

For word counts or TF-IDF weights, which are never negative, the result lies between 0 (no shared words) and 1 (the same direction). Because the vector lengths are divided out, it does not care how long a document is.

Jaccard similarity ignores counts and compares the two sets of words: the size of the intersection divided by the size of the union. It is simple, easy to explain and common for near-duplicate detection.

Euclidean distance is the ordinary straight-line distance between the two points, the square root of the summed squared differences. It is sensitive to length: a long document is far from a short one even if they say the same thing.

Comparing them on real vectors

The example below builds count vectors by hand for three short documents. A and B both talk about a cat; C is about stock prices. Then it tests an important property: a short text and the same text repeated five times.

Python Cosine, Jaccard and Euclidean side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first three lines rank the pairs the same way under all three measures: A and B are the closest (cosine 0.668), A and C share only the word on (0.158), and B and C share nothing at all (0.0). Note that Jaccard is lower than cosine for the first pair (0.286) because it counts only which words are shared (2 of 7 distinct words), not how often.

The last line shows the key difference. The short text and its five-fold repetition point in exactly the same direction, so cosine says 1.0, "identical topic". Euclidean distance says 5.657, "far apart", purely because the second vector is longer. For text, where documents differ wildly in length, cosine is almost always the better default.

Cosine can only look at words the two documents share. If two texts use different words for the same idea, car and automobile, cosine is 0. That is a limit of the bag-of-words representation, not of cosine itself.

Why TF-IDF makes similarity better

With raw counts, the and of dominate the dot product: two documents look alike because both are full of function words. With TF-IDF, those words have low weights, so the similarity reflects the informative words. Because TfidfVectorizer also normalizes each row to length 1, the dot product of two rows already is their cosine similarity, which makes the whole computation one matrix multiplication.

Scikit-learn offers cosine_similarity(A, B) for the full pairwise matrix, and cosine_distances for 1 - similarity.

Finding the nearest documents

A nearest-neighbour search computes the similarity between one vector (a query or a document) and all others, then sorts. The next example does two things with six short titles: find each document's closest partner, and rank everything against a query.

Python Nearest documents with TF-IDF

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each document's nearest neighbour is the one on the same topic: the two bread titles pair up (0.18), the two Python titles (0.50) and the two hiking titles (0.35). The ranked query results are more interesting. The best match is the bread document that contains the exact word baking (0.53), the Python document comes second only because it contains tips (0.31), and How to bake sourdough bread at home scores just 0.21 because it says bake, not baking. The bread document with bake should have ranked higher: cosine does not know that the two words share a stem. Stemming or lemmatization before vectorizing helps here.

Rows that are all zero, such as a document consisting only of stop words, have length 0, and dividing by zero gives undefined similarity. Scikit-learn returns 0 for those, but your own code has to guard against it.

Which measure to use

  • Cosine on TF-IDF: the default for text. Handles length differences and weights informative words.
  • Jaccard: when only presence matters, such as comparing sets of tags or detecting near-duplicates on shingles (n-grams).
  • Euclidean: rarely for sparse text vectors. It is common for dense vectors where length matters, though normalized dense vectors make it equivalent to cosine ordering.
  • Large collections: comparing a query against millions of documents one by one is slow. Real systems use an inverted index or an approximate nearest-neighbour index.

Application: plagiarism detection

Plagiarism detection is document similarity with a decision at the end. A typical pipeline has four steps: clean the text, turn each document into a vector, compare every suspect against the source, and report the pairs above a threshold. Two signals complement each other:

  • TF-IDF cosine similarity (from this lesson) measures overlap of vocabulary. It still scores a rewritten text well when the same important words appear, so it catches paraphrases that keep the key terms.
  • Shingle overlap compares sets of consecutive word n-grams (shingles), here word triples, with the Jaccard coefficient (size of the intersection divided by the size of the union). A verbatim copy shares almost every shingle, while a reworded text shares few, so a high value is strong evidence of copying.

A high cosine with a low Jaccard means "same topic and vocabulary, different wording": a case for a human to read. A high Jaccard means copied passages. If you have many documents, compute one TF-IDF matrix and its pairwise cosine matrix instead of comparing one pair at a time, and visualize the frequent terms of each file as a word cloud if you want a quick look at what two files share.

Python Plagiarism check: TF-IDF cosine plus 3-gram Jaccard

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exact copy scores 1.00 on both measures. The reworded text keeps the key words (photosynthesis, plants, oxygen), so the cosine is a substantial 0.57, but only 12% of its word triples match, so it is flagged for manual review rather than as copying. The unrelated text scores 0 on both. Notice that the thresholds are choices: tune them on documents you have labeled, and remember that quoted text, common phrases and references also create overlap.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Two documents point in exactly the same direction, but one is five times longer. What are their cosine similarity and Euclidean distance?

    Cosine divides out vector length, so same direction gives 1. Euclidean distance still sees the difference in length and stays above zero.
  2. 2. Which measure ignores how many times a word occurs and only looks at which words are present?

    Jaccard compares word sets, so repeated words do not change the score.
  3. 3. What is the cosine distance between two documents whose cosine similarity is 0.8?

    Cosine distance is defined as 1 minus cosine similarity, which is 0.2.
  4. 4. Why is TF-IDF usually better than raw counts for comparing documents?

    Common function words get low IDF weights, so the similarity score is driven by words that actually distinguish documents.
  5. 5. Two documents describe the same car using the words "car" and "automobile". What does cosine on bag of words give?

    Bag-of-words cosine only sees shared word forms. Different words for the same idea look unrelated, which is a limit of the representation.

Summary

  • Similarity measures compare document vectors; cosine, Jaccard and Euclidean are the common choices.
  • Cosine similarity is the angle between vectors: 0 for no shared words, 1 for the same direction, and independent of document length.
  • Jaccard compares word sets; Euclidean distance is sensitive to length and rarely suits sparse text.
  • Cosine distance is 1 - cosine similarity.
  • TF-IDF vectors with unit length make cosine a simple dot product and downweight common words.
  • Cosine on bag of words cannot see synonyms or word forms; stemming and embeddings help.

Next, put it all together in Building a Small Search Engine.