Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 39 ยท Storing and Cleaning Data

Deduplicating and Updating Data

Remove duplicates from scraped data with stable keys and fuzzy matching, detect new, changed and removed items between runs, and scrape incrementally.

Scrapers create duplicates in many ways. Paginated lists overlap when items are added mid-scrape. The same product appears in three categories. A restart re-fetches pages you already saved. Two sites list the same listing with slightly different titles. If you do not deal with duplicates, your counts are wrong, your averages are skewed, and your charts lie.

A second, related job is spotting change: which items are new since the last run, which changed price, which disappeared. That turns a one-off dump into a living dataset, and it is the basis of price trackers and monitors.

Pick a stable key

Everything starts with a key that identifies "the same thing" across runs. In order of preference:

  1. An ID the site gives you: a product ID, SKU, ISBN, listing number or a URL slug.
  2. The canonical URL, normalized (Building a crawler shows how).
  3. A hash of the fields that define identity, for example brand + model + size, lowercased and cleaned.

Avoid keys that change: titles are edited, URLs sometimes include a changing slug, and scraped positions on the page are meaningless. Parse the ID out of the URL if you must:

Python Extract a stable ID and drop exact duplicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Which duplicate to keep

When two rows share a key, decide which wins. Common rules: the most recently scraped, the one with the most filled fields, or merge them field by field (take the first non-empty value):

Python Merge duplicate records field by field

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Near-duplicates and fuzzy matching

When there is no ID, such as the same book listed on two sites, exact comparison fails: "The Hobbit (2nd ed.)" and "hobbit, the". First normalize (lowercase, strip punctuation, drop stop words), then compare similarity. The standard library's difflib is enough for modest data:

Python Near-duplicate detection with difflib

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A threshold like 0.85 is a starting point to tune by inspecting results. Do not blindly delete near-duplicates: flag them for review or merge only above a high threshold. For large datasets, comparing every pair is quadratic; group candidates by a cheap key first (first letter, brand, year), and use libraries such as rapidfuzz for speed.

Detect new, changed and removed items

Between two runs, compare the previous snapshot with the new one, both indexed by key:

Python Diff two snapshots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Set operations on dictionary keys do the heavy lifting. Be careful with removed: an item missing from one run may mean a failed request, not a deleted product. Mark an item as gone only after it has been missing for several runs, or when the site returns a clear 404 or 410 for it.

A fingerprint for page-level change

To know quickly whether a page changed at all, store a hash of the relevant content. Hash only the part you care about, not the whole HTML, which contains timestamps, tokens and ads that change every load:

Python Content hash to detect real changes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Incremental scraping

Re-scraping everything every time is wasteful and impolite. Cheaper strategies:

  • Sitemaps with lastmod: fetch only pages changed since your last run (robots.txt and sitemaps).
  • Conditional requests: send If-None-Match or If-Modified-Since and accept 304 Not Modified.
  • Newest-first listings: crawl a "latest" page and stop when you meet an ID you already have.
  • Tiered refresh: update popular or volatile items daily, the rest weekly.

The "stop at known item" rule in code:

Python Stop when you reach items you already have

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This only works for listings that are truly ordered newest first, so confirm that on the site before relying on it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes a good deduplication key?

    Titles and positions change; IDs and canonical URLs identify the same item across runs.
  2. 2. Why be careful before marking an item as removed when it is missing from one run?

    Absence in one run can be a scraping failure rather than a real deletion.
  3. 3. Why hash only selected fields rather than the whole page HTML?

    Hashing the relevant content avoids reporting change when only noise changed.
  4. 4. What is a safe way to handle fuzzy near-duplicates?

    Fuzzy matching produces false positives, so tune the threshold and check results.

Summary

  • Choose a stable key (site ID, canonical URL or hash of identity fields) before anything else.
  • Decide which duplicate wins, or merge field by field.
  • Normalize text and use similarity scores for near-duplicates, with review.
  • Diff snapshots with set operations to find added, changed and removed items; treat removals cautiously.
  • Fingerprint relevant content, and scrape incrementally with sitemaps, conditional requests and stop-at-known rules.

Next: Project: Book Catalog Scraper