Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 63 ยท Projects

Project: Movie Review Sentiment Analyzer

Build a movie review sentiment analyzer in Python: a labeled dataset, lexicon and TF-IDF baselines, grid search, negation handling and model interpretation.

Can a program tell whether a movie review is a rave or a pan? Sentiment analysis is one of the most common NLP tasks in industry, from product reviews to support tickets, and it makes a great second project: the labels are intuitive, you can read the data yourself, and the failure cases (negation, mixed opinions, sarcasm) teach you a lot about language.

In this project you assemble a small review dataset, build two baselines (a hand-made lexicon and a TF-IDF classifier), tune the classifier with a grid search, handle negation, and finally open the model to see what it learned. It builds on Sentiment Analysis, Logistic Regression for text and Evaluating Classifiers.

The plan

From reviews to a verdict

Dataset 240 generated reviews, half each way.
Lexicon A word list gives a first guess.
Classifier TF-IDF plus logistic regression.
Tuning Grid search picks the settings.

Finish by reading the learned weights.

1 / 2

Step 1: dataset, split and baselines

To keep everything small and runnable in your browser, we generate 240 short synthetic reviews with a seeded random generator (make_reviews). The recipe is visible in the code, so you know exactly what the model has to learn: 60 percent plain opinions ("The pacing was clever"), 30 percent negated ones ("The story was not dreadful", which is positive) and 10 percent with a "but" clause, where the part after "but" carries the verdict. Real corpora are messier and far larger, but the method is the same, and a generator lets you see precisely why each technique helps.

We keep 60 reviews aside as a test set that nothing may touch until the end. First baseline: a lexicon classifier that counts positive and negative words from a hand-made list (it knows 9 of the 12 adjectives on each side, like any real lexicon with gaps). It needs no training at all. Second baseline: TF-IDF features plus logistic regression.

Python Dataset, lexicon baseline and TF-IDF baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both baselines are weak: the lexicon scores 0.58 on the 60 test reviews and TF-IDF 0.60, only a little above a coin flip. The three example sentences show why. "The acting was wonderful" is easy. But "The acting was not wonderful" is wrongly called positive by both models, because they see wonderful and treat not as just another word. Roughly a third of our reviews are negated, so about a third of the data is being misread. (The lexicon also gets the "but" sentence wrong, for a different reason: it counts one positive and one negative word, calls it a tie, and ties become negative.)

This is a typical project moment. Your first model is poor, and the useful question is not "which fancier algorithm?" but "which kind of example is it getting wrong?"

TF-IDF plus logistic regression has settings that matter: whether to use only single words or also word pairs (ngram_range), how strongly to regularize (C), and whether to drop very rare words (min_df). Instead of guessing, GridSearchCV tries every combination, scores each with cross-validation on the training set only, and keeps the best. The test set is used once, at the very end.

Be cautious with small data: differences of a few points are noise, and a grid search will happily pick the luckiest setting. We use RepeatedStratifiedKFold, which repeats the cross-validation with different shuffles to steady the averages.

Python Grid search over n-grams, C and min_df

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The search tried 30 combinations (3 n-gram ranges, 2 min_df values, 5 values of C) with 12 cross-validation fits each. The best was word pairs (ngram_range=(1, 2)) with weak regularization (C=30), at 0.661 cross-validated accuracy, and the test set agrees at 0.68. The worst combination was single words with strong regularization at 0.569. Tuning bought about 9 points over the worst setting, which is useful but not transformative.

Why does the ceiling stay low? Pairs such as not wonderful are learnable, but there are twelve adjectives times several negation frames, and the model must see each combination often enough. Feature settings cannot fix a representation that lacks the idea of negation. We need a better feature, not a better knob.

Step 3: handle negation

The hard sentences from the first step showed the main weakness of bag-of-words models: not and good are separate features, so "not good" looks like a bit of both. A cheap, classic remedy is negation marking: after a negation word, attach a prefix like NOT_ to the following words until the next punctuation mark. Then not good becomes not NOT_good, a feature the model can learn is negative.

Here is the function on its own, with no dataset needed:

Python Negation marking

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scope matters: marking stops at a comma, so in the second example like becomes NOT_like but great after the comma stays positive. A stronger system uses a parser to find exactly which words a negation modifies; this rule gets most of the benefit for none of the complexity. It is also imperfect: "not only good but great" is positive, and the rule marks good as negated.

Step 4: interpret the model

A classifier you cannot inspect is a classifier you cannot trust. With a linear model, each feature has a weight: positive weights push toward positive sentiment, negative weights toward negative. Listing the extremes is the quickest sanity check, and it also shows whether the negation features helped. The final TryIt trains with and without negation marking, compares them on repeated cross-validation over all 240 reviews, and prints the strongest features.

Python Negation marking, comparison and top weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Negation marking changes the picture completely: mean cross-validated accuracy rises from 0.623 with plain words to 0.955 with negation marking, and the spread between folds shrinks. The model now sees NOT_wonderful as a feature of its own, so it needs far fewer examples of each combination.

The weights confirm what it learned. The strongest negative features are the negative adjectives (boring, messy, dreadful, ...) and, tellingly, NOT_moving. The strongest positive ones are the positive adjectives and NOT_disappointing. These are features no plain bag of words could contain.

The four test sentences then behave sensibly: "The acting was wonderful" gets 0.92 and "The acting was not wonderful" gets 0.18. "I would never call the plot boring" is correctly positive (0.81), because never triggers the marking too. The "but" sentence comes out at 0.54, a near coin flip. The model has no idea that the clause after "but" carries the verdict; it only sees that the sentence has one positive and one negative word. Fixing that needs word order, for example features for the last clause, or a sequence model such as the ones in Attention and the Transformer.

Step 5: what to say about the result

A good project report is honest about limits. For this one:

  • Data realism. Our reviews are synthetic and follow a few simple recipes, so 0.955 is far higher than any real review corpus would give. The numbers are good for comparing methods on the same data, not for promising a figure to a customer.
  • Domain. A model trained on movie reviews misreads other domains. In a laptop review, "unpredictable battery" is bad, while in a movie review "unpredictable plot" is praise. Retrain or at least re-evaluate for every domain, as covered in Evaluation, Bias and Responsible NLP.
  • Mixed and sarcastic reviews. "Great acting but a boring plot" has both signals; a single positive/negative label is the wrong target. Aspect-based sentiment, which scores acting and plot separately, is the next step up.
  • Calibration. A probability of 0.80 does not mean "right 80 percent of the time" unless you checked.

To go further, try the same pipeline with a pre-trained model from Pre-trained Language Models. Those models read whole sentences with context, so negation and mixed opinions come out much better, at the price of speed and size (see NLP in Production).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the test set kept untouched during the grid search?

    Settings are chosen using cross-validation on the training data. The test set is used once at the end, so the reported score is not tuned to it.
  2. 2. What problem does negation marking address?

    With separate features for "not" and "good", the model cannot represent "not good". Marking words after a negation creates features such as NOT_good that carry the flipped meaning.
  3. 3. On a small dataset, why be careful about picking the single best grid-search result?

    With so little data, cross-validated accuracies differ by random amounts. Treat near-ties as ties and prefer the simpler setting.
  4. 4. What do the largest positive and negative weights of a linear model tell you?

    In a linear model each feature has one weight, so the extremes show what the model relies on and expose accidental shortcuts.

Summary

  • A lexicon gives a no-training baseline; TF-IDF plus logistic regression is the standard learned baseline.
  • Tune with a grid search and cross-validation on the training set only, and score the test set once.
  • On small data treat small differences as noise and prefer simpler settings.
  • Negation marking (NOT_ prefixes up to the next punctuation mark) gives a bag-of-words model a way to represent "not good".
  • Inspect the learned weights and test sentences you wrote yourself.
  • Report the limits honestly: data size, domain, mixed opinions, sarcasm and calibration.

Next, build a different kind of system, one that finds documents rather than classifying them: Project: Document Search with Ranking.