Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 39 ยท Getting Started

Legal and Ethical Basics of Web Scraping

Understand the legal and ethical side of web scraping: robots.txt, terms of service, copyright, personal data, rate limits and a practical checklist before you scrape.

Whether scraping is "legal" has no single answer. It depends on what you collect, how you collect it, where you and the site are located, and what you do with the data. This lesson is not legal advice, and laws differ between countries, but it gives you the questions to ask and the habits that keep you on the right side of both the law and common decency.

A short summary: scraping public, non-personal facts gently and for a legitimate purpose is generally low risk. Hammering a server, bypassing access controls, copying creative work wholesale or harvesting personal information is where problems begin.

Questions to ask before you scrape

Before you scrape

Is it public? No login or paywall needed.
What does robots.txt say? Check what the site asks bots to avoid.
What do the terms say? Some sites forbid automated access.
Is it personal data? Privacy laws may apply.
1 / 2

robots.txt

Almost every site publishes a file at /robots.txt telling crawlers which paths they may visit. It is a convention called the Robots Exclusion Protocol. It is not a technical barrier and not a law by itself, but respecting it is the baseline of polite scraping, and ignoring it is strong evidence of bad faith if a dispute arises.

A typical file looks like this:

TEXT
User-agent: *
Disallow: /admin/
Disallow: /cart/
Crawl-delay: 5

User-agent: BadBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
  • User-agent names the bot a group of rules applies to; * means everyone.
  • Disallow lists path prefixes the bot should not request. An empty Disallow: allows everything.
  • Allow can make an exception inside a disallowed area.
  • Crawl-delay suggests seconds between requests (not every crawler honors it, but you should).
  • Sitemap points to a list of URLs the site wants indexed, which is a handy, polite place to find pages.

Python's standard library can read these rules. RobotFileParser.parse() accepts lines of text, so you can test it right here. On a real site you would call set_url("https://example.com/robots.txt") and read() instead.

Python Check a URL against robots.txt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how Allow: /admin/help carved an exception out of Disallow: /admin/. Real code should call can_fetch before every request and sleep for the crawl delay. Chapter 7 (Crawling, Politeness and Ethics) builds that into a crawler.

Terms of service

The terms of service (ToS) of a site are a contract you agree to by using it, or sometimes only by being told about it. Many terms forbid "automated access" or "scraping". Breaking a contract is a civil matter, and a site may block you, send a letter, or in some cases sue. Courts in different countries have reached different conclusions about when browsing terms are binding on someone who never logged in or clicked "I agree". Rules of thumb:

  • If you must log in or accept terms to see the data, the terms very likely apply to you. Be much more careful.
  • Data behind a login, paywall or technical barrier is a different category from data any visitor can see.
  • Do not circumvent technical protections (CAPTCHAs, IP bans, rate limits). Doing so can cross from "contract dispute" into computer misuse law in some countries.

Facts (a price, a date, a score) are generally not protected by copyright, but creative expression is: articles, reviews, photos, product descriptions. Copying and republishing them can infringe. In the European Union, a separate database right can protect a substantial collection of data even when each fact is free.

Safer uses: analysis, price comparison, research, and storing facts with a link back to the source. Riskier uses: republishing copied text or images, or building a competing product from someone's content.

Personal data

If a page shows names, emails, phone numbers, photos or anything that identifies a person, privacy law applies even if the information is public. Regulations such as the GDPR in Europe and the CCPA in California place duties on anyone who collects personal data: a lawful reason, limits on how long you keep it, security, and honoring deletion requests. If you do not need personal data, do not collect it. Drop fields you do not need as early as possible.

Be a good citizen of the web

Even when scraping is allowed, your script can harm a site. A server serving real customers has limited capacity, and a scraper that sends dozens of requests per second can slow it down or knock it over.

  • Rate limit yourself. One request per second or slower is a good default; follow Crawl-delay when given.
  • Identify yourself. A custom User-Agent with a contact address lets the owner reach you before blocking you.
  • Cache. Save pages you have already downloaded and re-use them while developing.
  • Prefer off-peak hours for large jobs.
  • Use the official channel when there is one. An API, a data export or an email asking for permission often gets you better data with no risk.
  • Back off on errors. If you see 429 (Too Many Requests) or 503, slow down instead of retrying harder.

A practical checklist

Python Pre-flight checklist

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This toy example fails on purpose: one unchecked item means you pause. If terms forbid automated access, ask the owner for permission or find another source.

When in doubt, especially for commercial projects, personal data, or content behind a login, get proper legal advice before you scrape. The cost of asking is tiny compared to the cost of a dispute.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is robots.txt?

    robots.txt is a convention for telling crawlers which parts of a site to avoid; it is a request, not a technical block, but polite scrapers always honor it.
  2. 2. Which data is generally the riskiest to republish?

    Creative text is protected by copyright, while simple facts generally are not.
  3. 3. A server answers 429 Too Many Requests. What should your scraper do?

    429 means you are going too fast. Back off; trying to disguise yourself or push harder is both impolite and risky.
  4. 4. Which is a good reason to collect as little personal data as possible?

    Laws such as GDPR attach obligations to personal data even when it is public, so collecting less reduces your legal exposure.

Summary

  • Legality depends on what you take, how, where, and why; this is not a substitute for legal advice.
  • Read and respect robots.txt, and check it in code with urllib.robotparser.
  • Terms of service are a contract; data behind logins and technical barriers needs extra care.
  • Facts are usually free to use, creative content is protected, and personal data brings privacy duties.
  • Rate limit, identify yourself, cache pages, back off on errors, and prefer an official API.

Next: HTTP Requests and Responses