Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 39 ยท Crawling, Politeness and Ethics

Legal and Ethical Scraping

An honest guide to the legality and ethics of web scraping: terms of service, copyright, personal data, access controls, and a practical checklist before you scrape.

"Is web scraping legal?" is the question everyone asks, and the honest answer is: it depends. Fetching publicly visible pages with a program is not illegal in itself in most places, but how you do it, what you collect and what you do with it can be. This lesson gives you a map of the issues and a practical way to decide. It is education, not legal advice: laws differ from country to country, they change, and courts have decided similar cases differently. If real money, personal data or a commercial product is involved, talk to a lawyer.

The questions that matter

Almost every scraping dispute comes down to a handful of questions. Ask them about every project.

  1. How do you get the data? Is it public, or behind a login, paywall or technical barrier?
  2. What do the site's rules say? Terms of service, robots.txt, API terms.
  3. What kind of data is it? Facts, creative content, or personal information?
  4. What do you do with it? Private analysis, a competing product, republishing, selling, training a model?
  5. What does your scraping do to the site? Load, cost, disruption.
flowchart TD
  A["Want to scrape a site"] --> B{"Official API or<br/>data download?"}
  B -- "yes" --> C["Use it. Read its terms."]
  B -- "no" --> D{"Behind a login or<br/>access control?"}
  D -- "yes" --> E["Stop unless you have permission"]
  D -- "no" --> F{"Personal data involved?"}
  F -- "yes" --> G["Privacy law applies:<br/>get advice or avoid"]
  F -- "no" --> H["Check robots.txt, terms<br/>and your intended use"]
  H --> I["Scrape politely"]

Terms of service

A website's terms of service (ToS) often forbid automated access. Whether a ToS can be enforced against someone who merely visited the site, never "signed up" and never clicked "I agree", is legally unsettled and varies by country. It is much stronger when you have an account, because creating one usually means you accepted the terms.

Practical rules:

  • Read the terms of any site you scrape regularly. Look for sections called "acceptable use", "automated access" or "scraping".
  • If they prohibit it, ask for permission or look for an API or licensed dataset. Many sites say yes, or have a paid data product.
  • A violation of the terms can still get you banned, sued for breach of contract, or both, even when no criminal law is involved.

Access and computer-misuse laws

Many countries have laws against unauthorized access to computer systems (in the US, the Computer Fraud and Abuse Act; the UK has the Computer Misuse Act; similar laws exist elsewhere). The core idea: accessing a system without authorization, or beyond what you are authorized to do, can be a crime.

In broad terms, courts have been more reluctant to treat scraping of genuinely public pages as unauthorized access than scraping that gets past a gate: a login, a paywall, an IP block you evade with rotating proxies after being told to stop, or a CAPTCHA you defeat. That is why this course keeps repeating the same advice: when a site clearly tries to stop automated access, do not engineer around it.

Facts, such as a price or a date, are generally not protected by copyright. Creative expression, such as articles, photos, reviews and descriptions, is. In some places (notably the EU) a database can have its own right when it took substantial investment to compile.

What this means in practice:

  • Extracting facts for analysis is usually on firmer ground than copying text and images.
  • Republishing someone's articles, photos or product descriptions is copyright infringement unless you have a licence or a legal exception.
  • Collecting a whole database of a competitor and using it in a competing product is risky in many jurisdictions.
  • Rules about text-and-data mining and AI training differ between countries and are evolving fast. Check the current rules and the site's stated opt-outs.

Personal data and privacy

Names, email addresses, phone numbers, photos, usernames, locations and anything else that can identify a person are personal data, even when posted publicly. Laws such as the EU/UK GDPR, California's CCPA and many others regulate collecting and processing it, often with requirements for a lawful basis, purpose limitation, security, and the right of people to access and delete their data.

Safe defaults:

  • Do not collect personal data you do not need. If your analysis needs only prices, do not store seller names.
  • Anonymize or aggregate early.
  • Never scrape private data, such as content behind friend-only settings or private groups.
  • Never use scraped contact details for unsolicited marketing without checking the rules for your country.
  • Store what you keep securely and delete it when you are done.

Courtesy and harm

Even if something is legal, it can still be wrong. Responsible scrapers:

  • Obey robots.txt and Crawl-delay (robots.txt and sitemaps).
  • Keep request rates low enough that a small site never notices (Rate limiting and retries).
  • Identify themselves with an honest User-Agent and a way to be contacted.
  • Cache so they do not request the same page twice.
  • Stop immediately when the owner asks.
  • Do not defeat CAPTCHAs, bot walls or other protections.
  • Give credit and respect licences when publishing results.

What scraping is typically fine for

Examples that tend to be low-risk when done politely: your own data exports; open-licensed and government open data; academic analysis of public facts with proper data handling; price comparison of public listings; archiving for personal use; monitoring changes to pages you own.

What tends to go wrong

  • Scraping behind a login against the terms.
  • Ignoring a cease-and-desist letter.
  • Copying large amounts of text or images and republishing them.
  • Harvesting personal data for marketing or profiling.
  • Hammering a small site until it falls over.
  • Circumventing blocks with proxy rotation after being told to stop.

A checklist you can run in code

The decision above is not a program, but a checklist you walk through before every new project is a good habit. Here it is as code, so your scraper project can carry its own record of the answers:

Python A pre-scrape checklist

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can also record the decision for later:

Python Keep a scrape log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A written note like this is useful if the site owner ever contacts you: it shows you acted in good faith.

When in doubt

  • Ask the site for permission, or for the data. A short, polite email works more often than you would expect.
  • Prefer licensed datasets and official APIs.
  • Scrape less, keep less, and publish only what you are sure you may.
  • Get legal advice for commercial use.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statement is most accurate?

    Several factors interact, and the rules vary by country, so there is no single yes or no.
  2. 2. Which is generally on firmer ground legally than the others?

    Facts are usually not protected by copyright, unlike creative text and images.
  3. 3. What should you do if a site shows a CAPTCHA or blocks your IP and tells you to stop?

    Circumventing protections after a refusal is what turns a dispute into a serious legal one.
  4. 4. Publicly posted names and email addresses are...

    Public availability does not remove privacy obligations.

Summary

  • Scraping is neither always legal nor always illegal; ask how, what, why and how much.
  • Prefer official APIs, read the terms, obey robots.txt and never get past login or bot-protection gates.
  • Facts are safer than copyrighted text and images; personal data triggers privacy law.
  • Be a good citizen: identify yourself, throttle, cache and stop when asked.
  • Keep a written record, and get legal advice for commercial projects.

Next: Cleaning Scraped Data