Skip to content
Tutorials Center
My learning

Topics

Lesson 94 of 111 ยท Projects

Project: Spam Filter

Build a spam filter in Python: generate a small message corpus, turn text into TF-IDF features, compare Naive Bayes and logistic regression, and tune the threshold.

A spam filter is the classic first text project because the whole machine learning workflow fits in one afternoon: messy input, a lopsided target, a model that is cheap to train, and a trade-off that has real consequences. A filter that lets spam through is annoying. A filter that hides a job offer or an invoice in the junk folder is much worse. Those two mistakes do not cost the same, and the interesting part of this project is deciding how to balance them.

You will build the filter in four steps on a small corpus that the code generates itself with a fixed seed. Real e-mail would need a download and a privacy review, and a generated corpus lets every number in this lesson match what you see when you press Run. The messages are simple word salads, so do not expect real-world accuracy from the final numbers. What carries over to real data is the method.

Four steps to a working filter

Corpus Messages plus a spam label.
Features TF-IDF turns words into numbers.
Models Naive Bayes against logistic regression.
Threshold Pick the cost balance you can live with.

Every step is one runnable block below.

1 / 2

Step 1: Build and inspect the corpus

The generator below writes messages from three word pools: words typical of spam, words typical of normal work mail, and neutral filler. A spam message draws about a third of its words from the spam pool, and a few from the work pool, so the classes overlap a little, as they do in real life. About a quarter of the messages are spam. That imbalance is deliberate, because real inboxes are lopsided too (see Imbalanced Classes).

We split with stratify so both parts keep the same spam share, and we fit nothing on the test part (Train, Validation and Test Sets). Then we build the baseline that every model has to beat: always predict "not spam".

Python Step 1: generate the corpus, split it, set a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline scores about 75% accuracy without ever flagging a single spam message, so its recall is zero. That is the accuracy trap from Classification Metrics: with a quarter of the mail being spam, "accuracy" alone cannot tell a useful filter from a useless one. From here on we report precision and recall as well, and we use average precision to compare models without picking a threshold first.

Step 2: Features and two models

A model cannot read text. TF-IDF turns every message into a vector with one number per vocabulary word: how often the word appears in the message, scaled down for words that appear in most messages. The TfidfVectorizer learns the vocabulary and the word weights from the training texts, so it belongs inside a pipeline together with the classifier. That way, cross-validation and the final test never leak vocabulary statistics from held-out messages.

We compare two classic text models. Naive Bayes is fast and works well with few examples. Logistic regression is a bit slower and usually gives better-behaved probabilities. We also give the logistic model word pairs (ngram_range=(1, 2)), so it can see phrases like "free cash".

Python Step 2: compare the baseline, Naive Bayes and logistic regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both real models land around 94% accuracy and an average precision close to 0.99, so on ranking quality they are nearly tied. Look at precision and recall instead: at the default threshold of 0.5 the logistic model flags no good mail at all (precision 1.000), but it also misses about a quarter of the spam (recall near 0.76). The 0.5 threshold is not a law of nature. It is just where the model's probability happens to cross one half, and the next step shows how to move it.

A very high average precision on a clean generated corpus is a sign that the data is easy, not that spam filtering is easy. Real filters fight adversaries who change their wording to dodge them, which is one reason production systems retrain constantly.

Step 3: Choose the threshold

predict applies a 0.5 cut-off. predict_proba gives the underlying probability, and you can cut anywhere you like (Thresholds, ROC and Precision-Recall Curves). Lowering the threshold catches more spam but starts flagging good mail. Raising it protects good mail but lets more spam through. The right point depends on what each mistake costs.

Suppose the team decides that losing a good message is five times worse than seeing a spam message. Then a sensible rule is to pick the threshold with the lowest total cost, where each false positive counts 5 and each false negative counts 1. The code sweeps thresholds, draws the precision-recall curve, and marks the cheapest one.

Python Step 3: sweep the threshold with a cost rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from top to bottom. At a threshold of 0.1 the filter catches every spam message but flags 128 good ones, so the cost explodes. At high thresholds good mail is safe, but recall collapses: at 0.8 only 9 messages are flagged. The cheapest points sit in the middle, where 0.4 and 0.5 tie at a cost of 18 (the code keeps the first one, 0.4), and the winner moves if you change the two cost numbers. That is the lesson to take away: the threshold is a business decision, and the model only supplies the probabilities.

We picked the threshold on the test set only to show the idea. In a real project, choose it on a validation set or with cross-validated predictions, then report the final numbers on a test set that played no part in the choice. Otherwise your reported precision is slightly optimistic.

Step 4: Open the model and test it

A spam filter you cannot inspect is a filter you cannot trust. Both models are linear in the word features, so we can read the words that push a message towards spam or towards normal mail. For logistic regression, those are the largest positive and negative coefficients. For Naive Bayes, we compare how likely each word is under each class.

Python Step 4: top words and a few hand-written messages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The top spam words are exactly the ones the generator put in the spam pool (deal, limited, cash, free, click, offer for the logistic model), and the most normal words come from the work pool. That is a useful sanity check, because it shows the model learned the real signal instead of an accident. On real data this check is where surprises appear: a word like a sender name or a date that correlates with spam in your training set but means nothing in general. If you find one, you have found leakage or sampling bias, and you should fix the data rather than trust the score.

Your hand-written messages show how the probabilities behave. The two spam-like messages score about 0.8 and the two work messages about 0.07, so the model separates them clearly, yet even the most spam-like message does not reach 1. Try mixing words from both pools in the editor and watch the probability slide into the uncertain middle, which is exactly where your threshold matters.

Where to go next

The same skeleton scales to real mail. Keep the pipeline, change the corpus, and add these upgrades one at a time, checking each against the baseline with cross-validation (Cross-Validation):

  • Character n-grams, which resist deliberate misspellings such as "fr33".
  • Extra hand-made features: the number of links, the share of capital letters, whether the sender is in the address book.
  • Calibration (see Probability Calibration and Uncertainty) if you need the probabilities themselves to be trustworthy.
  • Drift monitoring, because spammers change their language and a filter trained last year will quietly get worse.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. The baseline that always predicts 'not spam' scores about 75% accuracy here. What does that tell you?

    About a quarter of the messages are spam, so predicting 'ham' every time is right three times out of four while catching nothing. Any real model has to be judged on precision and recall, and compared with this baseline.
  2. 2. Why is TfidfVectorizer placed inside the same pipeline as the classifier?

    Fitting the vectorizer on all messages before splitting would let the held-out messages shape the vocabulary and the IDF weights. Inside a pipeline, every fit happens on the training part only.
  3. 3. A false positive (good mail sent to junk) is considered five times worse than a false negative. What should you do with the decision threshold?

    The threshold encodes how you trade the two mistakes. When one error is much more expensive, you pick the cut-off that minimizes the weighted cost on validation data instead of accepting the default.
  4. 4. The top 'spammy' words of your model include a sender's name that appears mostly in spam in the training data. What is the best conclusion?

    A feature that predicts the label only because of how the training set was collected will fail in production. Reading the top words is a cheap way to catch this kind of problem early.

Summary

  • A spam filter is a text classification problem with a lopsided target, so start with a majority-class baseline and judge models by precision, recall and average precision, not accuracy.
  • Put the TF-IDF vectorizer and the classifier in one pipeline so every step is fitted on training data only.
  • Naive Bayes and logistic regression are strong, cheap text models, and their weights can be read as words that push a message towards spam or ham.
  • The decision threshold is a separate choice from the model: pick it with a cost rule on validation data, then report test results once.
  • Inspect top words as a sanity check; surprising words signal leakage or sampling bias.

Next, switch from text to pictures in Project: Handwritten Digit Recognizer.