Skip to content
Tutorials Center
My learning

Topics

Lesson 56 of 63 ยท Projects

Project: Spam Filter

Build a spam filter end to end in Python: a labeled dataset, a Naive Bayes baseline, extra features, honest evaluation, error analysis and a safe threshold.

Spam filtering is the classic first NLP project, and for good reason. The task is clear, mistakes have an obvious cost, and every step of the usual workflow shows up: data, baseline, features, evaluation, error analysis, and a decision about trade-offs. In this project you build a complete filter with scikit-learn, and you will see why a single accuracy number is never the end of the story.

You need the ideas from Text Classification Overview, Naive Bayes, scikit-learn pipelines and Evaluating Classifiers. Here you combine them. Our dataset is small (60 original messages) so everything runs instantly in your browser; a real filter would use hundreds of thousands of messages, and the conclusions about method carry over.

The plan

Five steps, one project

Data Labeled spam and normal messages.
Baseline Bag of words plus Naive Bayes.
Features Add character n-grams and signals.
Evaluate Cross-validation, precision, recall.

Then read the mistakes and tune the threshold.

1 / 2

Step 1: data and a baseline

The dataset below has 30 spam and 30 normal ("ham") messages that we wrote ourselves. We made it hard on purpose: several normal messages contain spam-like words (free, urgent, call, won, a link), and the spam uses many styles (prizes, fake bank alerts, loans, fake parcels). A stratified split keeps the share of spam equal in the training and test sets. We then train the simplest sensible model and look at its confusion matrix.

Python Data, split and a Naive Bayes baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The split gives 42 training and 18 test messages, 9 of them spam. The confusion matrix reads: row = truth, column = prediction. The baseline makes one mistake of each kind, so accuracy, precision and recall are all 0.89 for both classes. That is a respectable start from five lines of model code, but eighteen messages are too few to say whether 0.89 is real, which is the point of the next step.

Note the two kinds of error in the matrix: one ham message was called spam (a false positive, the expensive kind) and one spam message got through (a false negative).

Step 2: better features and honest evaluation

A single 18-message test set is far too small to trust: one message changes accuracy by 5.6 points. When data is scarce, use stratified cross-validation instead: split five times, train on four fifths, test on the remaining fifth, and look at the average and the spread across the folds.

Now improve the features. Spam has a recognizable shape beyond its words:

  • character n-grams catch $1000, http://, !! and obfuscated words better than whole words,
  • hand-made signals such as the number of digits, exclamation marks, the share of capital letters and whether there is a link.

We combine them with FeatureUnion and compare five models with the same folds. Everything lives inside one pipeline, so cross-validation never leaks information from the test fold into training.

Python Cross-validated comparison of feature sets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table honestly:

  • Over five folds the plain Naive Bayes baseline reaches 0.967 accuracy, better than its 0.89 on the single split. Same model, different slice of data: that is how noisy a test set of 18 messages is.
  • With this little data the three solid models (Naive Bayes, words plus characters, and the signals at reduced weight) are within one message per fold of each other. The honest conclusion is "no clear winner yet", not "model 3 is best". With a few hundred thousand messages the richer features usually pull ahead.
  • Adding the hand-made signals at full weight is a disaster: accuracy drops to 0.78 with a huge spread between folds. Standardized numeric columns are dense and large, while each TF-IDF column is tiny and sparse, so the five signal columns drown out thousands of word features. Down-weighting them to 0.3 with transformer_weights fixes it.

That last lesson is easy to miss: when you mix feature families, check their scale. A feature that helps on its own can hurt in a union.

Step 3: look at the mistakes

Metrics tell you how often the model is wrong. Error analysis tells you why, and it is where most improvements come from. We train the final model on the training split, print every misclassified test message and the messages it was least sure about. We also turn the model's probability into a decision with a threshold. The default is 0.5, but when losing good mail is worse than letting spam through, you raise it and trade recall for precision.

The next example reuses the dataset and shows both. Because the final model is a linear one, we can also print the words that push a message most strongly toward spam or toward ham.

Python Error analysis and the decision threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this particular split the final model makes no mistakes at 0.5, so the useful list is the least certain messages. Two spam messages sit close to the line: the fake antivirus warning (0.52) and the "win a free tablet" offer (0.57). Both are short and use few of the classic spam words. The most doubtful ham message (0.36) is "Sorry I missed your call", which shares the word call with spam. These are exactly the cases to look at when you collect more training data.

The threshold table shows the trade-off in numbers. At 0.3 every spam is caught (recall 1.00) but some good mail is flagged too (precision 0.82). At 0.5 both are perfect on these 18 messages. At 0.7 precision stays at 1.00 but recall collapses to 0.44, because this tiny model is rarely very confident, so a high threshold lets most spam through. Your real choice depends on data size: a model trained on far more messages is more confident and tolerates a higher threshold.

The top words make sense: now, free, cheap and today push toward spam, while ordinary conversation words such as me, we and at push toward ham. Always check the list for accidents. A word like to among the spammy words is a small warning that the model is also learning style, not only meaning.

The last two lines show why we keep spam-like words in the ham data. "Free tickets to the match, call me when you are free!" contains free twice and still gets only 0.19, while a loud message with a link, a dollar amount and capitals gets 1.00.

Step 4: use the lessons

The project teaches lessons you will meet again on real data:

  • Start with a baseline. Bag of words plus Naive Bayes is a few lines of code and often hard to beat by much. Every later improvement is measured against it.
  • Evaluate with the right metric. In spam filtering, a false alarm (good mail in the spam folder) costs more than a missed spam, so precision on the spam class matters more than raw accuracy, especially because real inboxes are imbalanced.
  • Use cross-validation on small data. One lucky or unlucky split can fool you.
  • Read the errors. They suggest new features (links, money, capitals), new data to collect (more ham that uses spam-like words) and sometimes label mistakes.
  • Choose the threshold from the cost of mistakes, not from habit.
  • Expect an arms race. Spammers adapt, so plan for monitoring and retraining.

The step that matters most before release is a fresh test set that you never touched while tuning. Our threshold experiment looked at the test split, which is fine for teaching but would leak information in a real project: choose the threshold on a validation set, and report final numbers on a separate test set.

Be wary of datasets where spam differs from ham in some accidental way, such as spam all being longer or all coming from one source. The model will happily learn that shortcut and fail in real use. Check what the top features are, as the example above does.

Ideas for extending the project

  • Add a third class for "promotions" and use multi-class classification.
  • Replace the Naive Bayes with a linear SVM and compare, using the same folds.
  • Build a feature that checks whether a link's domain contains digits or hyphens.
  • Write the cleaning step as a reusable pipeline function and add tests for it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do we use a stratified split on spam data?

    Stratification preserves class proportions in each part, so the small test set is not accidentally missing one class.
  2. 2. For a spam filter, which error is usually worse?

    Losing real mail can cost a job or a payment, so precision on the spam class is usually prioritized; a missed spam is a smaller annoyance.
  3. 3. What does raising the decision threshold from 0.5 to 0.9 typically do?

    A higher bar means only very confident messages are called spam, so fewer false alarms (higher precision) but more missed spam (lower recall).
  4. 4. Why put all feature extraction inside one pipeline when cross-validating?

    If the vectorizer is fitted on all the data first, information from the test fold leaks into training. A pipeline refits everything per fold.

Summary

  • A complete NLP project follows data, baseline, features, honest evaluation, error analysis and a threshold decision.
  • Stratified splits and cross-validation give trustworthy numbers on small datasets.
  • Character n-grams and hand-made signals (digits, exclamation marks, capitals, links) capture the shape of spam.
  • Error analysis and top-feature inspection reveal what the model learned and which data is missing.
  • The threshold encodes the cost of each kind of mistake; choose it on validation data.

Next, apply the same workflow to opinions in Project: Movie Review Sentiment Analyzer.