Skip to content
Tutorials Center
My learning

Topics

Lesson 138 of 149 ยท Projects

Project: Sentiment Analysis

Build a sentiment classifier in Python. Clean text, count words, vectorize reviews with TF-IDF, train logistic regression with scikit-learn, evaluate it, find its mistakes and wrap it in a function.

Every online shop, app store and restaurant site collects text reviews. Nobody can read thousands of them, so companies ask a program to sort them: is this review positive or negative? That task is called sentiment analysis, and it is a classic first project in machine learning. In this project you'll build a small sentiment classifier for cafรฉ-app reviews, from raw text to a function you can call on any sentence.

You'll use string methods, regular expressions, dictionaries and random first, then the tools from Machine Learning with scikit-learn. The project has five stages. Each one runs on its own. If you want a larger, more detailed treatment afterward, the NLP course has a movie review sentiment project with tuning and a deeper look at negation.

Stage 1: the data

A real project would load thousands of real reviews from a file (see CSV files). To keep everything inside the page, we generate 400 labeled reviews with a small function. It combines a topic (coffee, service, staff...) with a positive or negative adjective and a sentence template, and sometimes adds a neutral filler sentence. About 15% of the reviews contain a negation: "not terrible" is a positive review, "not great" a negative one. Negation is what makes sentiment analysis tricky, and the project will show you how it trips up simple methods.

The generator is seeded, so everyone gets the same data. Each later example repeats the function so it runs on its own. On your own machine you'd keep it in one module and import it.

Python Stage 1: generate and inspect the data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The labels are 1 for positive and 0 for negative, the convention scikit-learn expects. The classes are nearly balanced, so a classifier that always guessed the majority class would score around 50%. That baseline is what any real model has to beat.

Stage 2: clean the text and build a word-list baseline

Computers see "Great!", "great" and "GREAT" as three different strings. Text cleaning makes them the same: lowercase everything, drop punctuation, and split the text into tokens (words). A short regular expression does the job.

With clean tokens we can try the simplest sentiment method of all, a lexicon: count the known-good words and the known-bad words in a review and compare. It uses no machine learning, so it makes a useful baseline. Our lexicon is typical of a hand-written one: it knows most, but not all, of the adjectives.

Python Stage 2: tokenize and score with a word list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mistakes fall into two groups, and both are instructive. Reviews that use one of the three adjectives the lexicon doesn't know score zero, so they default to "negative". And reviews with a negation such as "not terrible" get the opposite of the right answer, because a bag of single words cannot see the word "not" attached to the word after it. A hand-built list also costs effort to write and keep up to date. A machine-learning model learns the word weights itself from labeled examples.

Stage 3: vectorize and train with scikit-learn

A model cannot read words, only numbers. Vectorizing turns each review into a row of numbers. The most common recipe is TF-IDF: each column is one word of the vocabulary, and each value says how often that word appears in this review (term frequency), scaled down for words that appear in nearly every review (inverse document frequency), because they carry little meaning. The result is a table with one row per review and one column per word.

On those rows we train a logistic regression classifier and test it on reviews it has never seen. A Pipeline bundles the vectorizer and the classifier, so the vocabulary is learned from the training data only.

Python Stage 3: a TF-IDF and logistic regression pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the confusion matrix like this: the top-left number counts negative reviews correctly called negative, the bottom-right counts positive ones correctly called positive, and the other two cells are the mistakes. The report adds precision (when the model says positive, how often is it right?) and recall (of the truly positive reviews, how many did it find?). Compare the accuracy with the lexicon from stage 2: the model has learned all twenty adjectives, including the ones the word list missed. Negation still hurts it, as we'll see.

Stage 4: a fairer score, and what the model learned

A single train/test split gives one number that depends on which reviews landed in the test set. Cross-validation splits the data into folds (here five), trains on four and tests on the fifth, rotates, and reports all five scores. Every review is tested exactly once, so luck matters less.

We compare two settings of the vectorizer. By default it only counts single words. With ngram_range=(1, 2) it also counts bigrams, pairs of neighboring words such as "not great", which gives the model a way to learn that "not" followed by a positive word is negative. Then we read the learned coefficients: a positive weight pushes a review toward "positive", a negative weight toward "negative".

Python Stage 4: cross-validation, bigrams and top words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bigrams lift the cross-validated accuracy from about 81% to about 91%, because phrases like "not perfect" now get weights of their own. Look at the lists of top words: single words such as "horrible", "dirty" and "fantastic" appear, but so do bigrams such as "not rude" (strongly positive) and "not fantastic" (strongly negative). The model has learned to flip the meaning after "not", exactly what the single-word model could not do. Reading the weights is how you check that a model learned something sensible rather than an accident of the data. If a word like "saturday" showed up near the top, you would suspect that the dataset, not the language, was driving the result.

Be careful about what this experiment can claim. The data are generated from a small vocabulary and a few templates, so the scores here are far tidier than real reviews would give. Treat the project as a map of the workflow, not as a promise of 91% accuracy on real text.

Stage 5: find the mistakes, then wrap it in a function

Accuracy hides which reviews fail. Looking at the mistakes is the most useful debugging step in machine learning. Then we package the trained model into a function that returns a label and a confidence, and that admits when it has nothing to go on.

Python Stage 5: errors, confidence and a predict function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error list shows the pattern: 30 of the 36 misclassified reviews contain "not", usually a negation of an adjective whose "not ..." pair appears only a few times in the data. More examples of those pairs would fix most of them, which is a typical lesson: look at the mistakes to learn what data is missing.

Two design choices here come from real-world experience. First, the function returns a confidence and says "unsure" below a threshold, because a wrong answer delivered confidently is worse than an honest "I don't know". Second, text made entirely of unknown words gets no guess at all, since the model has nothing to base one on. In an actual product you would send the "unsure" reviews to a human.

To reuse the trained model without retraining, save it with pickle (or joblib) and load it in your app. Only unpickle files you created yourself, because loading a pickle can run arbitrary code.

Ideas to extend it

  1. Real data. Collect a few thousand real labeled reviews in a CSV file and compare the cross-validation scores with the generated ones. Expect lower numbers.
  2. Three classes. Add "neutral", or turn star ratings into labels, and use the same pipeline.
  3. Better cleaning. Remove stop words, add stemming or lemmatization (see the NLP course) and compare scores.
  4. Other models. Replace logistic regression with MultinomialNB (Naive Bayes) or a linear SVM and compare with cross-validation.
  5. A tiny app. Put predict_sentiment behind a command-line prompt, or behind an API like the one in Project: JSON API with Flask.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does TF-IDF do to a review?

    TF-IDF gives a model numeric features. Words that appear everywhere get little weight, while distinctive words get more.
  2. 2. Why put the vectorizer and the classifier in one Pipeline?

    Fitting the vectorizer on all the data would let information from the test set influence training. A pipeline refits both steps correctly on each training split.
  3. 3. Why does the word-list (lexicon) method get 'The coffee was not great' wrong?

    A bag of single words ignores word order, so 'not' and 'great' are scored separately. Bigrams give the model a way to learn the pair.
  4. 4. What does ngram_range=(1, 2) add to the features?

    Bigrams let the model give 'not great' its own weight instead of treating 'not' and 'great' separately, which helps with simple negation.
  5. 5. Why does predict_sentiment return unsure for low-confidence results?

    Real systems route uncertain predictions to review. A threshold on the predicted probability decides which ones.

Summary

  • Sentiment analysis is text classification: decide whether a text is positive or negative. Always compare against a baseline such as guessing the majority class or a simple word list.
  • Clean and tokenize text first: lowercase, remove punctuation, split into words with a regular expression.
  • Vectorize with TF-IDF so a model sees numbers, then train a logistic regression classifier; wrap both in a Pipeline.
  • Judge it with a confusion matrix, precision and recall, and cross-validation. A single small test split is too noisy.
  • Read the learned weights and the misclassified examples to see whether the model learned something sensible. Bigrams help with phrases such as "not great".
  • Return a confidence and admit uncertainty. Generated data like this teaches the workflow; real reviews are messier and need far more examples.

Next: Project: Inventory Manager, where you'll model a shop's stock with classes and save it as JSON.