Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 39 ยท Text and Web Mining

Text Mining Basics

Turn text into data: tokenization, stop words, stemming, bag of words, n-grams, TF-IDF, cosine similarity, and document classification and clustering with runnable Python examples.

Most of the world's information is text: emails, reviews, news, support tickets, social posts. Text mining (text analytics) extracts useful patterns from it. The core difficulty is that algorithms such as classifiers and clustering need numbers, while text is a sequence of words with grammar, ambiguity and spelling variation. The craft of text mining is turning text into a good numeric table.

The text mining pipeline

Collect Documents, reviews, tickets
Clean Tokenize, lowercase, stop words
Vectorize Counts or TF-IDF
Mine Classify, cluster, search

Each step changes what the model can learn.

1 / 2

Preprocessing

  1. Tokenization: split text into units, usually words.
  2. Normalization: lowercase, strip punctuation and accents.
  3. Stop word removal: drop very common words (the, is, of) that carry little meaning. Not always wise: not flips sentiment.
  4. Stemming or lemmatization: reduce running, runs, ran to a common form. A stemmer chops suffixes by rules (running becomes run, fast but crude); a lemmatizer uses a dictionary and part of speech (better becomes good).
Python A small preprocessing function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the crude stemmer: batteries becomes batteri. A stem need not be a real word, only consistent. Libraries such as NLTK or spaCy provide better stemmers and lemmatizers.

Bag of words

The bag-of-words model represents a document by how often each word occurs, ignoring order. All documents together define a vocabulary; each document becomes a vector with one count per vocabulary word. The matrix of vectors is the document-term matrix.

Python Document-term matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bag of words loses word order: "dog bites man" and "man bites dog" get identical vectors. N-grams recover a little context by counting word pairs (bigrams) or triples: ngram_range=(1, 2) adds "cheap flights" or "data mining" as features, at the cost of a much bigger vocabulary.

TF-IDF

Raw counts overweight words that appear everywhere. TF-IDF (term frequency times inverse document frequency) weighs a word higher when it is frequent in this document but rare in the collection:

tfidf(t, d) = tf(t, d) * log(N / df(t))

where N is the number of documents and df(t) the number containing the term. A word in every document gets weight near 0; a distinctive word gets a large weight. scikit-learn uses a smoothed variant and normalizes each vector to length 1.

Python TF-IDF by hand, then with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Similarity between documents

Because TF-IDF vectors are normalized, the cosine similarity (the cosine of the angle between two vectors) is just their dot product: 1 means the same direction, 0 means no shared words. This powers search ("rank documents by similarity to the query"), duplicate detection and recommendations.

Python A tiny search engine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Classifying and clustering documents

Once documents are TF-IDF vectors, any classifier from this course works; linear models and Naive Bayes are strong for text because the vectors are very high-dimensional and sparse.

Python Topic classification and clustering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The clusters separate the sport headlines from the finance ones even though no labels were used (ARI 1.0 means a perfect match). A deeper tour of text features is in scikit-learn text feature extraction.

Beyond bag of words

  • Word embeddings (word2vec, GloVe) map words to dense vectors where similar words are close; transformer models create context-aware embeddings.
  • Named entity recognition finds people, places and organizations.
  • Information extraction pulls structured facts (prices, dates) from text.
  • Topic models and sentiment analysis are covered in the next lesson.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the bag-of-words model discard?

    Only counts per word are kept, so different sentences with the same words look identical.
  2. 2. Which word gets the lowest TF-IDF weight?

    IDF is log(N/df); when df = N the idf is 0, so common words are down-weighted.
  3. 3. What is the main difference between stemming and lemmatization?

    Stemmers are fast and crude; lemmatizers return real dictionary forms.
  4. 4. What does a cosine similarity of 0 between two TF-IDF vectors mean?

    With non-negative weights a cosine of 0 means no overlap in terms.

Summary

  • Text mining converts text into numeric vectors, then applies classification, clustering or search.
  • Preprocessing (tokenizing, stop words, stemming) shapes the vocabulary; n-grams add a little word order.
  • TF-IDF down-weights common words; cosine similarity compares documents.
  • Linear classifiers and k-means work well on TF-IDF vectors.

Next: Web Mining and PageRank