Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 39 ยท Storing and Cleaning Data

Cleaning Scraped Data

Turn messy scraped strings into clean values: whitespace, prices and currencies, dates, numbers, units, missing values and validation, with runnable Python examples.

Scraped data arrives dirty. A price is " $1,299.00\n", a rating is "4.5 out of 5 stars", a date is "3 days ago", and one product has no price at all. If you store those strings as they are, every later step (sorting, charts, comparisons) breaks in small, annoying ways. Cleaning is the step that turns text from a page into values a program can trust.

The rule of thumb: scrape raw, clean separately. Keep the original text next to the cleaned value while you develop, so you can see what a rule did and fix it when the site changes. The functions in this lesson are small, pure and easy to test, which is exactly what you want.

Whitespace and text

HTML is full of line breaks, indentation and non-breaking spaces (\xa0). get_text(strip=True) helps, but not enough on its own:

Python Normalize whitespace and Unicode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Zero-width characters such as โ€‹ are not removed by \s; if they appear, strip them explicitly with s.replace("โ€‹", "").

Prices and currencies

Prices come as $1,299.00, 1.299,00 โ‚ฌ, EUR 89,90, From ยฃ5, or Free. The tricky part is the decimal separator: in much of Europe the comma is the decimal mark and the dot the thousands separator. Decide from the data: if the last separator in the digits is followed by exactly one or two digits, it is the decimal mark.

Python Parse prices with different formats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two professional habits are visible here. Use Decimal (or integer cents) for money, never float, to avoid 0.1 + 0.2 surprises. And return None instead of guessing when the text does not look like a price, so bad rows can be counted and inspected.

Numbers, ratings and units

Pull the number out of the sentence with a regular expression and convert it:

Python Extract numbers and units

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dates and relative times

Pages show dates as March 3, 2026, 03/04/2026, 2026-03-03T10:00:00Z, or 3 days ago. Prefer machine-readable sources when they exist: <time datetime="2026-03-03"> or JSON-LD. Otherwise parse with explicit formats, and be careful with ambiguous 03/04/2026 (March 4 or April 3?). Decide from the site's locale and test on days above 12.

Python Parse absolute and relative dates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always inject today as a parameter rather than calling datetime.now() inside the function; it makes the function testable and deterministic. Store dates in ISO format (YYYY-MM-DD) and timestamps in UTC.

Missing values and validation

Decide what a missing value means. Is None "the page does not show a price" or "our selector broke"? Two defenses:

  1. Validate each record against simple rules and collect the problems instead of crashing.
  2. Track the missing rate per field. If the share of rows without a price suddenly jumps from 2% to 90%, the site layout changed.
Python Validate records and report missing rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Cleaning many rows with pandas

When the data is tabular, pandas does the same cleaning in bulk. Vectorized string methods apply a rule to a whole column:

Python Bulk cleaning with pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

errors="coerce" turns unparseable values into NaN instead of raising, which is exactly what you want while cleaning, as long as you then count them.

Keep the raw value

A good table keeps both: price_raw ("1.299,00 โ‚ฌ") and price (1299.00). When a parser has a bug, you can re-run it over the raw column without re-scraping. Storing raw HTML pages on disk is even better, because then you can fix the extraction without touching the site again (Rate limiting and retries explains caching).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use Decimal or integer cents for prices instead of float?

    Binary floats cannot represent many decimal fractions exactly; money should use exact types.
  2. 2. What should a price parser return for text that is not a price?

    Returning None avoids inventing data and lets you measure how often parsing fails.
  3. 3. Why pass today into a date parser instead of calling datetime.now() inside?

    Injecting the current date lets you test relative dates like '3 days ago' with fixed results.
  4. 4. Why keep the raw scraped value next to the cleaned one?

    Parsers have bugs and sites change; the raw text lets you recompute.

Summary

  • Scrape raw, then clean in small pure functions that return None when unsure.
  • Normalize whitespace and Unicode; parse prices with Decimal and careful separator handling.
  • Extract numbers and units with regex; use explicit date formats and inject today for relative dates.
  • Validate every record, and monitor missing rates to detect broken selectors.
  • pandas cleans whole columns at once; keep the raw value for re-processing.

Next: Saving to CSV, JSON and SQLite