Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 63 ยท Cleaning and Tokenization

Stop Words and When to Keep Them

Learn what stop words are, how to remove them with a list or a frequency cutoff, and why blind removal breaks negation, phrases and modern neural models.

Words like the, of, and and is appear in almost every English sentence. They carry grammar, but on their own they say little about the topic of a document. NLP practitioners call such words stop words, and for decades the standard advice was to throw them away before counting anything.

That advice is sometimes right and sometimes harmful. In this lesson you will remove stop words with a hand-made list, build a list from data instead of trusting a generic one, and then see the classic way removal goes wrong: deleting the word not. You need a tokenizer first, so skim tokenization basics if you have not seen one.

What stop words are for

Why people drop stop words

Smaller data Fewer tokens to store and count.
Sharper topics Content words stand out in counts.
Faster models Smaller vocabularies train quicker.
1 / 2

There is no official list. Libraries ship lists of about 100 to 400 English words, and they disagree. Whether not, no, never, against or up belong is a judgment call, so treat any list as a starting point that you inspect.

Removing stop words with a list

The simplest method is a Python set. Sets give constant-time lookups, so filtering a million tokens is fast. Tokenize, lowercase, and keep only the tokens that are not in the set.

Python Filtering with a stop-word set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sentence shrinks from 14 tokens to 6, and what remains still hints at the topic: library, corner, open, staff, friendly, everyone. This is the effect that made stop-word removal popular for search and topic counting.

Always lowercase before the lookup, or add both cases to the list. Otherwise The at the start of a sentence slips through while the is removed.

Building a list from your own data

A generic list may not fit your domain. In a corpus of cooking recipes, add and cup appear everywhere; in legal text it is shall and hereby. A data-driven approach counts in how many documents each word appears. Words found in nearly every document cannot help tell documents apart, so they are good candidates for removal.

Document frequency (DF) is better than raw count here, because one long document repeating a word does not make it a stop word. This idea is the heart of TF-IDF, where such words are down-weighted automatically instead of deleted.

Python A frequency-based stop list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the list that comes out: add, and, the and until. the and and are on every generic list, but add and until are recipe-specific filler that no general list would contain. The same output also shows the weakness of a pure frequency cutoff: in, a and for are still in the filtered document, because they appear in fewer than half of these six documents. The threshold is a dial. With a very small corpus like this one, a cutoff of 0.5 is aggressive; on a real corpus you would pick something like 0.8 to 0.95 and review the list by hand before using it.

You can also go the other way and remove rare words (appearing in only one or two documents), which are often typos or names. The min_df and max_df options of scikit-learn vectorizers do exactly this, as covered in feature hashing and large vocabularies.

The negation trap

Here is the failure that surprises beginners. Many generic lists contain not, no, nor and never, and these words flip the meaning of a sentence. After blind removal, a negative review can look positive.

Python When removal flips the meaning

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the generic list, "This movie was good" and "This movie was not good" both become ['movie', 'good']: two opposite reviews become identical. Keeping the negation words preserves the difference. A common refinement is to attach the negation to the following word (not_good) so the model sees one feature; sentiment analysis shows this trick.

Other things stop-word lists damage:

  • Phrases and names: The Who, Take That and to be or not to be vanish or turn into fragments.
  • Questions and intent: who, what, where and how define a question. Removing them hurts question answering and chatbots.
  • Short documents: a search query like to be or not to be or the office becomes empty after removal.
  • Languages: a list for English is wrong for Spanish or German. Pick one per language.

When to remove, when to keep

A practical rule of thumb:

Task Recommendation
Word clouds, topic counts, keyword extraction Remove stop words
Bag of words with TF-IDF for topic classification Often fine either way; test both
Sentiment analysis Keep negations, maybe more
Question answering, chatbots, translation Keep everything
Transformers and other neural language models Keep everything; they use word order and function words

For most classifiers the honest answer is to measure. Run your pipeline with and without removal and compare the held-out score, as in evaluating classifiers. If the numbers match, prefer the simpler pipeline that keeps all words, because it has fewer decisions to maintain.

In libraries

NLTK and spaCy both ship lists; they are read-only here because the browser runner has neither installed.

Python
from nltk.corpus import stopwords
english = set(stopwords.words("english"))   # close to 200 words, includes "not"

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The staff are friendly")
[t.text for t in doc if not t.is_stop]

scikit-learn takes a stop_words argument in its vectorizers (stop_words="english"), which uses its own list. Since these lists differ in size and content, look at the actual words before trusting them; scikit-learn's documentation itself warns about problems with its list.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is document frequency preferred over total word count for building a stop list from data?

    Document frequency counts each word once per document, so a word qualifies only if it is spread across many documents.
  2. 2. What goes wrong if the stop list contains "not" and you remove stop words from sentiment data?

    With "not" removed, a negative and a positive review can reduce to the same content words, so a classifier cannot tell them apart.
  3. 3. For which task is keeping all words, stop words included, the safest default?

    Neural language models use function words and word order, so removing them throws away information they depend on.
  4. 4. Why should you lowercase text before filtering with a stop-word set?

    Set lookups are exact. Without lowercasing, "The" at the start of a sentence would not match "the" in the list.

Summary

  • Stop words are very common function words such as the and of; there is no single official list.
  • Remove them with a set after lowercasing, or build a list from document frequency in your own corpus.
  • Blind removal can delete negations and question words and so change the meaning.
  • Keep stop words for sentiment, question answering, translation and neural models.
  • Decide by measuring: compare scores with and without removal.

Next, reduce words to their base forms in Stemming and Lemmatization.