Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 39 ยท Projects and Practice

Project: Spam Filter and Topic Explorer

A text mining project: build a spam filter with TF-IDF and Naive Bayes, inspect the words that drive it, evaluate with precision and recall, then explore unlabeled messages with topics.

This project applies text mining, classification and topic modeling to one realistic problem: a mailbox owner wants unwanted messages filtered, and also wants to understand what the legitimate messages are about. You will build a spam classifier, inspect what it learned, check its errors and then explore the non-spam messages with topics.

To keep everything runnable, the messages are generated from word templates. Real text is messier, but the workflow is the same. Each block is self-contained.

Step 1: Build a labeled corpus

Python Generate and inspect messages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 30% of normal messages contain one spammy word ("free lunch"), and every spam message also contains two ordinary words: realistic overlap that makes the task non-trivial.

Step 2: Train and evaluate

Split first, vectorize inside a pipeline so the vocabulary and IDF are learned from training data only, and score with precision and recall for the spam class. For a mail filter the costly error is a false positive: a real message lost to spam.

Python Naive Bayes spam filter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inspecting the learned log-ratio per word is an easy, honest check that the model learned sensible things and not artifacts, for example a sender name that only appears in the training spam.

Step 3: Look at the errors and tune the threshold

Predicted probabilities let you trade false positives against missed spam. Raise the spam threshold when losing a legitimate email is worse than seeing some spam.

Python Errors and threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 4: Explore the legitimate mail with topics

After filtering, the owner wants an overview of what the remaining messages are about. No labels exist for topics, so use NMF on TF-IDF vectors and read the top words of each topic.

Python Topic explorer for ham messages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two topics read as "work" and "social". With real mail you would try several topic counts and name the topics yourself.

Extensions

  • Add bigrams and character n-grams to catch obfuscated words ("fr33").
  • Replace Naive Bayes with logistic regression and compare in cross-validation.
  • Add features beyond words: message length, number of links, share of capital letters.
  • Retrain periodically: spammers change their vocabulary (concept drift).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a spam filter, which error is usually more costly?

    Losing a real message can cost more than seeing some spam, so the threshold is often set conservatively.
  2. 2. Why vectorize inside a pipeline fitted on the training data only?

    Fitting the vectorizer on all data lets test information influence the features.
  3. 3. How can you check what a Naive Bayes spam model learned?

    Words with the largest log-ratio are the strongest spam indicators, a quick sanity check on the model.
  4. 4. Why are the topics of the legitimate mail found with NMF unlabeled?

    NMF discovers word groups from the data itself; humans interpret and name them.

Summary

  • Build text classifiers with a TF-IDF plus Naive Bayes pipeline and evaluate precision and recall for the class that matters.
  • Inspect learned word weights and misclassified messages to find problems; tune the decision threshold to the cost of each error.
  • Combine supervised filtering with unsupervised topic discovery to understand what remains.

Next: Practice Exercises