Skip to content
Tutorials Center
My learning

Topics

Lesson 101 of 111 ยท Projects

Project: Fake News and SMS Spam Detection

Two text classification projects in one: detect SMS spam and fake news with TF-IDF and linear models, handle disguised words, and catch a source leak that fools the score.

Text classification is one of the most common first machine learning jobs. This project builds two classifiers with the same toolkit (TF-IDF features and linear models) and learns something different from each. The SMS spam filter teaches you to read the errors a model makes, and to defend it against spammers who disguise their words. The fake news detector teaches the most important lesson in text modeling: a model can score 95% for the wrong reason, and the score collapses the moment the data comes from somewhere new.

You have already built a small spam filter with threshold choice, so this project goes after what that one did not cover: model comparison on a harder corpus, error analysis, character n-grams, and leakage. As before, the code generates its own data with fixed seeds. Real data sets would need downloads and a privacy review, and generated text lets every number below match what you see when you press Run. The text is a toy, so the absolute scores say nothing about real SMS or news. The method and the pitfalls carry over.

Two projects, one toolkit

SMS spam Short, messy, adversarial.
Fake news Longer, and easy to fool yourself with.
TF-IDF Words become weighted numbers.
Linear models Fast, strong, readable.

Same pipeline, different traps.

1 / 2

Part 1: SMS spam

Step 1: Compare models and read the errors

The generator writes short messages from templates: prize offers, fake bank alerts and delivery scams for spam, and ordinary chat for normal messages ("ham"). To keep it realistic, some normal messages use spammy words ("are you free friday? call me when you can") and about 2% of the labels are wrong, as in any real labeled set. About 20% of the messages are spam.

We compare a baseline that always says "ham" with three standard text models, all on TF-IDF features: Naive Bayes, logistic regression and a linear support vector machine (linear SVM). Features are fitted inside a pipeline on the training part only. After the table, the code lists the messages the logistic model gets wrong.

Python SMS: baseline versus three models, then the errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline reaches 78.8% accuracy while catching no spam at all, which is the accuracy trap again. The three real models tie at 98% accuracy, 0.98 precision and 0.925 recall. A three-way tie to the last digit is a clue worth chasing, and the error list explains it. Of 750 test messages, only 15 are wrong, and most of them are label noise: "meeting moved to sunday, same room" is labeled spam, and a message that begins "Congratulations you have been selected" is labeled ham. The model is right and the label is wrong. When all your models fail on the same rows, suspect the data before you suspect the model. Chasing a higher score here would mean teaching the model to reproduce mislabeled examples, so the sensible move is to fix or drop those labels, and then to report the remaining ceiling.

Step 2: Spammers fight back

Real spammers write "fr33" and "w1n" to slip past a filter that has only seen the normal spelling. The next experiment trains on clean spam and tests on disguised spam, where most letters in long words are replaced by look-alike digits. Compare three feature sets. Word features need the exact word. Character n-grams (analyzer="char_wb", sequences of 2 to 5 characters inside words) match fragments of words, so "c4rd" still shares pieces with "card". Finally, a few hand-made features that disguise cannot hide: how many digits the message contains, whether it has a link, the share of capital letters, and the length.

Python SMS: word features, character n-grams and hand-made features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The disguise matters. Word features lose a lot, with recall falling to 0.774, because the words they learned ("customer", "checking") are not there in disguised form. Character n-grams recover most of it (0.932), since pieces of the disguised words still match. Combining plain words and n-grams in one model (0.842) is worse than n-grams alone here: the exact-word features give the model patterns that the disguise breaks, so adding them did not help. Adding the hand-made features to the character model reaches 0.954 on this generated data, where disguised words do contain many digits. On real messages the gain would differ, but the lesson holds: when someone is actively trying to beat your model, choose features that are hard to fake, and test on data that looks like the attack, not like your training set.

Part 2: Fake news

Step 3: A score that is too good

Fake news detection is a binary classification of articles: 1 means fake, 0 means real. Our generated news has three topics (economy, health, tech) and the usual signs: fake articles use somewhat more sensational words ("shocking", "exposed", "hoax"), and real ones use a few of them too, so the difference is modest. There is one more thing, which mirrors a famous trap in public fake-news data sets: the source of the real articles puts a dateline tag at the start of 90% of its stories, like WASHINGTON (Wire) - . The fake articles come from other sources and have no tag. That tag has nothing to do with truth, but it is a perfect-looking clue.

The program trains a TF-IDF and logistic regression model, prints its top features, and then runs the experiments that expose the leak: evaluate on a second source that has no tags, and retrain with the tag removed.

Python Fake news: the dateline leak, step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the four result lines in order.

  1. Same source, test split: 95.5% accuracy. This looks great, and it is the number most tutorials would stop at.
  2. The top features give it away. The model's strongest "real" features are wire, ottawa wire and berlin wire, the tag. It learned to recognize the publisher, not to judge truth. The "fake" side has the sensational words you would hope for, but the tag is what carries the score.
  3. A new source without the tag: 71.9% accuracy, 0.651 precision. Real articles from the other publisher have no tag, so the model labels many of them fake. Recall of fake articles stays high (0.952) partly because so many real articles are wrongly called fake. The dataset's score did not measure the skill we care about.
  4. With the tag stripped before training: 82.5% on the same source and 82.8% on the new source. The score is honest now, lower, and it stays the same when the source changes. The remaining signal is the sensational vocabulary, and a modest 82% is what that signal is worth.

The same thing happens with real data. Public fake-news sets are often built from a few outlets, so the publisher name, a boilerplate footer, a photo credit or an author tag can reveal the label. This is leakage: a feature that exists in the training data but does not carry the real signal.

An honest fake-news model needs more than word counts. Real systems use many sources, split train and test by source (so a publisher never appears in both), and combine language with other evidence such as claims that can be checked. A language model alone can recognize a style, not the truth of a statement.

A checklist for text classification projects

  • Start with a baseline (the majority class) and use precision, recall and F1, not accuracy alone, whenever the classes are unbalanced.
  • Keep all fitted steps in a pipeline, so the vocabulary and IDF weights come from the training part only.
  • Read the errors. If all models fail on the same rows, look for label noise or an ambiguous task.
  • Read the weights. The top positive and negative features show what the model relies on, and leaks stand out.
  • Split the way production will. If new data will come from new sources, authors or time periods, hold out whole sources, authors or periods.
  • Plan for adversaries. Character n-grams and hand-made features make evasion harder, and monitoring catches the rest (monitoring and drift).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Three different models all score exactly the same accuracy and recall on the SMS test set. What is the best first step?

    When very different models make the same mistakes, the cause is usually in the data, such as wrong labels. The errors list showed spam-labeled chat and ham-labeled prize messages.
  2. 2. Why do character n-grams help against disguised words like 'c4rd' and 'n33ds'?

    A word feature needs the exact word. Character n-grams break each word into overlapping pieces, so a few swapped characters only change some of the pieces.
  3. 3. The fake-news model's top 'real' features are 'wire' and 'ottawa wire'. What does this tell you?

    The tag appears mostly in real articles from one source and has no connection to truth. A model that relies on it looks good on the same source and fails on a new one.
  4. 4. How should you split data to check whether a fake-news model works on publishers it has never seen?

    A random split puts articles from every publisher on both sides, so publisher-specific leaks still help. Holding out whole sources measures how the model behaves on new ones.

Summary

  • TF-IDF plus a linear model (logistic regression, linear SVM or Naive Bayes) is a strong, fast baseline for text, but judge it against a majority-class baseline with precision, recall and F1.
  • Error analysis can show that the remaining mistakes are label noise, which sets a ceiling on any model.
  • Character n-grams and features such as digit counts and link flags help against disguised words.
  • A feature that identifies the data source (a publisher tag, a footer) can produce a high score that disappears on new sources; read the top features and test on held-out sources.
  • Stripping the leak lowers the score to an honest number, which is a better result than an impressive wrong one.

Next, forecast numbers instead of labels in Project: Sales and Demand Forecasting.