Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 63 ยท Cleaning and Tokenization

Cleaning and Normalizing Text

Clean raw text for NLP with Python: lowercasing, accents, punctuation, whitespace, HTML, URLs and emoji, a regex pipeline, and the pitfalls of over-cleaning.

Real text arrives dirty. It has HTML tags left over from web pages, URLs, email addresses, doubled spaces, odd encodings, emoji, shouting in capitals and punctuation in all the wrong places. A model that sees "Great!", "great" and "GREAT!!!" as three unrelated words wastes data and splits its counts. Cleaning and normalizing text maps these variants to a smaller set of forms.

Cleaning is also where beginners do the most damage. Every step throws information away, and the information you discard might be exactly what the task needs. This lesson shows the standard operations, builds them into a small regex pipeline, and ends with the habit that matters most: decide each step based on your task, and check what it removed.

Common cleaning steps

Case Lowercase so Great equals great.
Markup Strip HTML tags and entities.
Links Replace URLs and emails with a placeholder.
Whitespace Collapse runs of spaces and newlines.
1 / 2

Normalizing characters first

Do character-level work before anything else, because later steps assume stable characters. Apply Unicode normalization (usually NFKC, see Levels of Language) so that the ligature, the full-width letter and the composed or decomposed accent all collapse to one spelling. Then handle case.

For case, lower() is fine for English; for comparisons across languages prefer casefold(). Remember that case sometimes matters: "US" the country, "us" the pronoun, "Apple" the company. Lowercasing is a trade, not a free lunch, and many modern neural models keep the original case.

Accent stripping, covered in the previous lesson with an NFD trick, should be switched on only when the task benefits, such as search. Do not strip accents blindly in a Spanish or German classifier.

Whitespace and punctuation

Whitespace is the safest thing to normalize. re.sub(r"\s+", " ", text).strip() replaces every run of spaces, tabs and newlines with one space and trims the ends. Do it last in your pipeline, because earlier steps tend to leave extra spaces behind.

Punctuation is a more careful decision. Removing everything that is not a letter looks tidy, but look at what it does to real strings:

Python What blind punctuation removal does

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

can't becomes cant, 3.5% becomes 35, $4.99 becomes 499, and C++ and C# becomes C and C, which loses the difference between two programming languages and the letter. A gentler rule keeps apostrophes inside words and treats most other punctuation as a separator by replacing it with a space (not with nothing), so e-mail becomes two tokens and not the invented word email.

Removing HTML, URLs, emails and emoji

Text scraped from the web is full of structure. Strip tags with a simple regex (fine for cleaning snippets; for whole pages use a real HTML parser) and decode entities such as & with html.unescape. URLs and email addresses rarely help a topic or sentiment model, and they inflate the vocabulary with millions of one-off strings. A good compromise is to replace them with a placeholder token such as URL or EMAIL, which keeps the signal that a link was present while removing the unique detail.

Emoji deserve thought. For sentiment and social-media tasks they are often the strongest evidence in the text. Options are to keep them as tokens, to map them to words with a small dictionary, or to remove them when the task ignores them.

A regex cleaning pipeline

Putting this together, each step is a small function that takes a string and returns a string. Order matters: remove HTML before URLs (so a tag does not swallow a link), and URLs before punctuation (so the link is gone before its punctuation is stripped).

Python Cleaning a messy post, step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the final line: cafรฉs bars visit url or mail email loved it. The page markup, the link and the address are reduced to the placeholders url and email, the & is decoded, the punctuation is gone, and the whitespace is tidy. The emoji disappeared too, because [^\w\s'] treats them as punctuation. If the task were sentiment analysis, that would be a bug, and the next example shows a fix.

Keeping the emoji signal

Instead of deleting emoji, map the ones you care about to words before the punctuation step. A small dictionary is enough for a demonstration (real projects use larger tables or a library).

Python Turning emoji into words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Now the emoji survive as ordinary tokens (emoji_happy, emoji_angry) that a counting model can use. Notice that the clean function is just the earlier pieces composed: Building a Reusable Preprocessing Pipeline turns that idea into a configurable tool.

The pitfalls of over-cleaning

A few rules of thumb prevent most damage.

  • Look at the data after every step. Print ten examples before and after. Bugs in cleaning are silent: the program still runs, the numbers just get worse.
  • Match cleaning to the model. Classic bag-of-words models benefit from heavy normalization. Pre-trained transformers were trained on natural text with case and punctuation, and often do better when you clean lightly. Their tokenizers also handle most of what you would clean by hand.
  • Keep the raw text. Store the original next to the cleaned version so you can change your mind later.
  • Be careful with negation and emphasis. Capitals, repeated letters ("sooo good") and exclamation marks carry sentiment. Dropping the word "not" is worse still; Stop Words and When to Keep Them explains why.
  • Apply the same cleaning at training and prediction time. A model trained on lowercased text and served raw text will quietly misbehave.

Treat cleaning choices as hyperparameters. Run your classifier with and without lowercasing, accent stripping or number removal, and keep the choices that improve a held-out score, not the ones that look tidiest.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is it better to replace punctuation with a space than with nothing when cleaning 'e-mail me'?

    Deleting the hyphen glues the parts together into a word that may not exist. A space separates them safely.
  2. 2. Why should HTML tags be stripped before URLs and punctuation?

    Cleaning steps interact. If you stripped punctuation first, the angle brackets would already be gone and the tag names would remain as ordinary words.
  3. 3. For a sentiment classifier, which of these cleaning decisions is most likely to hurt?

    Emoji and exclamation marks are strong sentiment signals. Whitespace, entities and links are usually safe to normalize.
  4. 4. A model is trained on lowercased text. At prediction time you pass raw mixed-case text. What is the likely result?

    Training and serving must use the same cleaning. Otherwise tokens like 'Great' are unknown to a vocabulary that only contains 'great'.

Summary

  • Cleaning maps messy variants of the same thing to one form, which shrinks the vocabulary and pools counts.
  • Normalize Unicode first, then case, then handle markup, links, emoji and punctuation, and collapse whitespace last.
  • Replace punctuation with spaces, keep apostrophes inside words, and use placeholders such as URL and EMAIL instead of deleting detail you might need.
  • Emoji, capitals and exclamation marks often carry sentiment; convert or keep them when the task needs them.
  • Inspect output after every step, keep the raw text, and apply identical cleaning in training and production.

Next: Tokenization: Splitting Text into Words