Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 39 ยท Crawling, Politeness and Ethics

robots.txt and Sitemaps

Read and obey robots.txt with urllib.robotparser, honor Crawl-delay, and use XML sitemaps to discover pages the polite way. Runnable Python examples.

Before you fetch a single page from a site, there are two small files worth reading. robots.txt tells automated clients which parts of the site the owner would like them to leave alone, and the sitemap lists the pages the owner wants found. Together they are the site's own instructions to scrapers, and a good scraper starts with both.

robots.txt is a convention, not a lock: nothing technically stops you from ignoring it. But ignoring it is exactly how scrapers get banned, blocked at the network level and, in the worst cases, taken to court. Honoring it is the minimum standard of professional behavior, and it is built into the Python standard library.

Where it lives and what it contains

The file is always at the root of the host: https://example.com/robots.txt. It is a list of groups. Each group begins with one or more User-agent lines, followed by rules:

TEXT
User-agent: *
Allow: /admin/public/
Disallow: /admin/
Disallow: /cart
Crawl-delay: 5

User-agent: BadBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
  • User-agent names the robot the group applies to; * means everyone not matched by a more specific group.
  • Disallow blocks a path prefix; an empty Disallow: allows everything.
  • Allow carves exceptions out of a Disallow.
  • Crawl-delay asks for seconds between requests. It is not part of the official standard and not every crawler supports it, but it is a clear request you should honor.
  • Sitemap points to sitemap files (this line is independent of any group).

Paths match from the start of the URL path, * matches any characters, and $ anchors the end. When rules conflict, the most specific (longest) matching rule wins.

Checking permission in Python

urllib.robotparser implements all of this. On a live site you call rp.set_url(...) then rp.read(). For testing, feed it lines directly:

Python Check URLs against robots.txt rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the /admin/public/ rule in this example: Allow: /admin/public/ is more specific than Disallow: /admin/, so the allowed path wins. Python's parser applies the first matching rule in file order, and it does not pick the longest match like Google's crawler does. That is why the Allow exception is written before the broader Disallow; if you swapped them, /admin/public/help would come out disallowed here.

In a real scraper wrap it once and reuse it:

Python
import time
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser

USER_AGENT = "PriceWatchBot"
_parsers = {}

def allowed(url):
    parts = urlsplit(url)
    root = f"{parts.scheme}://{parts.netloc}"
    rp = _parsers.get(root)
    if rp is None:
        rp = RobotFileParser(root + "/robots.txt")
        rp.read()                       # fetches over the network
        _parsers[root] = rp
    return rp.can_fetch(USER_AGENT, url)

If robots.txt returns 404, the site has no restrictions, and the parser treats everything as allowed. If it returns 401 or 403, the parser treats everything as disallowed. If the server is down (5xx), be careful: wait and retry rather than assuming permission.

robots.txt is about crawling politeness. It is not a license. Permission to fetch a page does not mean you may republish its content, and a "Disallow" does not become fine just because the page is also reachable some other way. Read Legal and ethical scraping.

Meta robots and X-Robots-Tag

Individual pages can carry their own instructions: <meta name="robots" content="noindex, nofollow"> in the HTML or an X-Robots-Tag response header. These mainly control search-engine indexing, but they signal intent. A site that says noindex, nofollow on a page is not inviting you to harvest it. Check for them when you build a crawler, and treat nofollow as "do not follow the links on this page".

Sitemaps: the owner's own list of pages

A sitemap is an XML file listing URLs, often with last-modified dates. It is the polite way to discover pages: instead of crawling link by link, you read the list the site publishes. It also tells you which pages changed, so you can re-fetch only those.

Python Read a sitemap with the standard library

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Big sites use a sitemap index: a file whose root element is <sitemapindex> and whose entries are <sitemap><loc>... pointing to other sitemap files (each limited to 50,000 URLs). Some are gzip-compressed (.xml.gz). Handle all three cases:

Python Sitemap index and filtering by date

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dates in ISO format compare correctly as strings, which keeps the filter simple. In a crawler you would fetch each child sitemap politely (with delays) and process it the same way.

Finding sitemaps

  1. The Sitemap: lines in robots.txt (the parser's site_maps() returns them).
  2. Common names: /sitemap.xml, /sitemap_index.xml.
  3. The site's search-engine settings or footer.

A polite startup routine

flowchart TD
  A["Start with a site"] --> B["Fetch robots.txt"]
  B --> C["Read Crawl-delay and Sitemap lines"]
  C --> D["Fetch sitemap (politely)"]
  D --> E["Filter URLs with can_fetch"]
  E --> F["Fetch pages with delay"]

Every URL passes the can_fetch check, even the ones that came from the sitemap.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Where is robots.txt located?

    robots.txt is always served from the root path of the host.
  2. 2. What does Crawl-delay: 5 ask crawlers to do?

    Crawl-delay is a request for the number of seconds between successive requests, honored by polite crawlers.
  3. 3. Which standard-library module parses robots.txt?

    RobotFileParser implements can_fetch, crawl_delay and site_maps.
  4. 4. What is a sitemap index?

    Large sites split their URLs over many sitemaps and list them in a sitemapindex file.
  5. 5. If robots.txt allows a URL, can you freely republish its content?

    Permission to fetch does not grant rights to reuse the content.

Summary

  • Fetch and honor robots.txt before crawling; urllib.robotparser does the matching, Crawl-delay and Sitemap lookup.
  • It is a courtesy convention, not a license or a security measure.
  • Pages may also say noindex or nofollow via meta tags or headers; treat these as intent.
  • Sitemaps (and sitemap indexes) list pages and last-modified dates: the polite way to discover and refresh content.
  • Check can_fetch for every URL, including ones from sitemaps.

Next: Building a Crawler